Filter out logs using props.conf and transfors.conf

[daniel_augustyn]

I am pulling logs from the firewalls via scripts on a heavy forwarder (via scrips from the app for Checkpoint). How to create props.conf and transfoms.conf to filter some logs from being indexed by the indexers. And where to put them? In the $Splunk/etc/apps/APP_NAME/local folder or in the $SPLUNK/etc/system/local/ folder on the heavy forwarder?

This is what I've got so far and it doesn't seem to be picking up the logs that I want to filter out.

props.conf:
[source::...opsec]
sourcetype = opsec

[opsec]
TRANSFORMS-set= setnull, setparsing

transforms.conf
[setnull]
REGEX = LAB
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = .
DEST_KEY = queue
FORMAT = indexQueue

[masonmorales · Accepted solution]

I believe sourcetype renaming is only applied at search time, so the [opsec] stanza in props.conf would not be picked up during the parsing phase.

What happens if you change your props.conf to:
[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

If you still have problems, try adding the config to your indexers too. To rename the sourcetype, add a props.conf to your search head(s):
[source::...opsec]
sourcetype = opsec

[daniel_augustyn]

It started picking up after I had deleted these two files and created new ones. And after I rebooted the heavy forwarder. I still don't know what was the issue at the first place, since the files look identical and I was rebooting Splunk before after each change.

[daniel_augustyn]

It still doesn't seem to be picking up the events with "LAB" word in them. Is there something wrong with the code in any of these files?

[Jeremiah]

You can put your props.conf and transforms.conf in an app or under system/local. The system/local directory will win out over anything you have set it an app.

http://docs.splunk.com/Documentation/Splunk/latest/admin/Wheretofindtheconfigurationfiles

Using an app is generally a good idea, because it allows you to package and re-deploy it if you need to. Some people will put all of their index-time props and transforms in a single app, others break them up by technology or application. Thats more a matter of preference and what works best for you.

For the settings you have here, I would do one of two things. Either set the sourcetype in your inputs.conf file so you don't have to set it in your props.conf, or move your TRANSFORM to your source stanza:

[source::...opsec]
sourcetype = opsec
TRANSFORMS-set= setnull, setparsing

[daniel_augustyn]

The regex doesn't seem to be picking up the events I want to filter out: REGEX = LAB

I am still getting all of the event with "LAB" word indexed.

[Jeremiah]

Actually looking at what you have, since you want to drop events with "LAB", you just need the setnull transform, not the setparsing.

TRANSFORMS-set = setnull

You can see something similar here:

https://answers.splunk.com/answers/107605/filtering-events-out-via-props-conf-and-transforms-conf.ht...
and here
https://answers.splunk.com/answers/293599/how-to-configure-propsconf-and-transformsconf-to-f-2.html

[daniel_augustyn]

Still doesn't pick up the events I want to filter out. Is this something off with this:
[source::...opsec]
sourcetype = opsec