regex to cull particular dest and src IP's before indexing

[dthompsonsplunk]

Hi All,
I have an issue where I need to cull certain IP sources and destinations from syslog sources before it gets indexed as these fill up our quota and we don't need to see it. I'm pretty sure I have the regex right even through its probably a but long winded but I've never used regex before.
What I need to do is put this in the transforms.conf file on the indexer but I'm not sure on the context I need to have so Splunk recognises it and culls it.
I have copied in what I have below, which isn't working, props file first then transforms. Its a Light Forwarder running in a deployment server environment

/opt/splunkforwarder/etc/system/local# cat props.conf
[loadbalancer]
NO_BINARY_CHECK = 1
pulldown_type = 1
TRANSFORMS-null = loadbalancer_setnull

/opt/splunkforwarder/etc/system/local# cat transforms.conf
[loadbalancer_setnull]

REGEX = dest_ip=\"172\.16\.100\.(6[5-9]|[7-8][0-9]|9[0-5])|224\.0\.0\.(5|18)|192\.168\.18[6-7]\.(0|16|48|240)\".*source_ip=\"172\.16\.\d+.\d+|0\.0\.0\.0\"

DEST_KEY = queue
FORMAT = nullQueue

there are a fair few combos we need to cull but 2 examples are:

dest 224.0.0.18 src 172.16.189.105
dest 192.168.187.48 src 0.0.0.0

Have been trying to get it work in the search app by prefixing with |rex field=_raw or |rex field=dest_ip and various combos but cannot get that working either.

[spurushothaman]

How can i exclude entries with src=0.0.0.0 ?

[dthompsonsplunk · Accepted solution]

Just coming back to add a bit of info for anyone who may have come across this themselves. I have now got this working after working on it off and on when I had spare time.

• We use Light Forwarders so these will not do any parsing or data manipulation so the regex needed to go on the Indexers.
• We have a clustered environment so the transforms and props file in here: /opt/splunk/etc/system/local do nothing. btool showed them being integrated into the configs but nothing worked. I had to make my changes to transforms and props here: /opt/splunk/etc/slave-apps/_cluster/local before I could get my reg ex working to drop particular logs before it went into Splunk.
• This is the regex I am now using and is working. The hardest thing was trying to find the proper syntax for my regex beginning. ie this bit REGEX = dest_ip=( REGEX = dest_ip=(\"172\.16\.(148|250|181|190)\..*\"|\"10\.10\.\d+.\d+\"|\"224\.0\.0\.(5|18)\"|\"192\.168\.(196|197)\.(0|16|48|240)\").*source_ip=(\"172\.26\.\d+.\d+\"|\"0\.0\.0\.0\")

Hope this helps someone in my similar situation

[jeffland]

If you want to discard certain events, the regex you set in transforms.conf does not need any capturing groups. It simply looks at the event and if it finds the regex, the event goes to the place you define (in your case nullQueue). Therefore, if you want to exclude ips with "dest"-values in your local network from 192.168.0.x to 192.168.2.x, you would set

REGEX = dest 192\.168\.[0-2]\.\d{1,3}

Now, depending on the combinations you want to exclude, these expressions can get quite complicated, but this should get you started. Perhaps as an idea to work with, it is sometimes easier to exclude a whole lot of events and then re-add them to the indexing queue. Your props.conf would then look something like this:

TRANSFORMS-set= setnull,setparsing

With setnull being the stanza in transforms.conf that sends events to nullqueue and setparsing another stanza which sends events to indexQueue instead of nullQueue.

[dthompsonsplunk]

Thanks very much jeffland this has pointed me in the right direction I have also since found out I need to move this off the forwarder to the indexer as the forwarder wont do this type of search.
Just as an aside, I can't seem to locate the transforms and props folders in the /home/splunk/idx01-configurations/etc/system/local# directory on the Indexer. Do I need to create these from scratch?

[jeffland]

If the settings of a particular .conf file haven't been used so far, the file won't be there and you need to create it. But take care where you place your configuration, see here for information on that. To see which configuration is applied in the running system, you can use btool.

[dthompsonsplunk]

Awesome, btool b just what I needed. I have no idea which files are being used on the indexer, theres over 20 transform and props files that turned up in my search results and none were where I expected them to be so I will need to create these myself in the right spot. Thanks very much for your help.

[jeffland]

You're welcome 🙂