Getting Data In

Why are my props and transforms.conf not filtering data on the heavy forwarder?

prakash007
Builder

I have a Heavy Forwarder installed which sends the logs to Splunk Cloud. Here's the workflow, please shed some light on this....
UFs-----> HWF ----->Splunk clould indexers

-data is flowing into the cloud and searchable, but when I tried to place some filtering on the HWF using props.conf and transforms.conf, it's not filtering.
-I want to drop the events which contain keyword POST...

Here're the configs I have used...

HWF=========
outputs.conf

indexAndFoward = false (not sure if I have to make it true for filtering and i don't want to index data locally/ filtering didn't work though even when it's true)

props.conf

[source::/path/to/my/apache/logs/access*.log]    or  I also tried with sourcetype in here [apache_access]
TRANSFORMS-null = filter_diagnose, filter_scalars

transforms.conf

[filter_diagnose]
REGEX =`\"POST\s\/.*diagnoseMonitor`
DEST_KEY = queue
FORMAT = nullQueue

[filter_scalar]
REGEX = `\"POST\`s\/.*`Scalar.html
DEST_KEY = queue
FORMAT = nullQueue

=====================================================================
I have also tried this way using key words to filter.

props.conf

[source::/path/to/my/apache/logs/access*.log]    or  I also tried with sourcetype in here [apache_access]
TRANSFORMS-set= setnull, setparsing

transforms.conf

[setnull]
REGEX = POST
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

MuS
Legend

Hi mcnamara,

most likely the class names for TRANSFORMS- in props.conf are not unique; from the docs http://docs.splunk.com/Documentation/Splunk/6.3.1511/Admin/Propsconf :

TRANSFORMS-<class> = <transform_stanza_name>, <transform_stanza_name2>,...
* <class> is a unique literal string that identifies the namespace of the field you're extracting.

Try to change the set and null class to some longer stings like SetNullQueueForTheSourceFooBecauseWeDontNeedThem

Hope this helps ...

cheers, MuS

PS: remember to restart Splunk after the changes and also remember it will only apply to new events coming in.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...