clarification with indexing time and trasforfms.conf needed

[damucka]

Hello,

I would like to reduce the license consumption and therefore think of installing HF and applying filtering there.
However, the more I read about it the more I come to the conclusion it will cost me some considerable amount of work as we have quite some number of clients. Also, it looks like the HF is nothing else but Splunk Enterprise, which instead of indexing does transform/ filtering and then forwards the data to the indexer.

So, I am asking myself if it would not be better to skip the HF idea and create the filtering with the help of props.conf / transforms.conf directly on the target indexer, e.g. like that:

props.conf
...
[(?::){0}*hanatraces]
TRANSFORMS-hatracesFilterEvents = hanatracessetnull


transforms.conf
...
[hanatracessetnull]
REGEX=(?m).*i TraceContext     TraceContext\.cpp.*\s|(?m).*e ExprConversionTo ConvertExpression\.cpp.*\s|(?m).*STATS_WORKER.*\s
DEST_KEY=queue
FORMAT=nullQueue

Which in this case would filter out the events including corresponding patterns I do not need.
But it would only make sense if the above transforms takes place before indexing/license calculation.
Could anyone confirm it?

Kind Regards,
Kamil

[gcusello · Accepted solution]

Hi @damucka,
yes your idea is correct: HF is a complete Splunk Enterprise installation configured to transform events and send them to Indexers.
It's usefun in many Use Cases where I need to concentrate logs (e.g. many Universal Forwarders in a segregate network that I don't want to open to Indexers).

If the load of your indexers isn't too high (or the resources aren't too few) you can filter data in Indexers without problems: I usually do it, I use HFs only when I need to concentrate data, never only for filtering or transforming.

About the license consuption, filteriring is applied before indexing and license calculation, so you haven't license consuption from the deleted logs.
You'll have only a little overload on the Indexers (CPU): use the suggested configurations for Indexers and monitor load and you'll haven't any problem!

Ciao.
Giuseppe

[broberg]

First of all read
https://docs.splunk.com/Documentation/Splunk/8.0.1/Forwarding/Routeandfilterdatad

If your data have not been going trough any processing first then you can use props/transforms on your indexer and route events to null queue if you do not want it to be calculated to your licensing. (Same as you can use splunk on a single machine that can take all roles, but it is not recomended for enterprise installations)