Splunk Search

Using the transforms.conf file to only forward events that match a regex.

JordanPeterson
Path Finder

I've got a log file that get's 2 different event formats depending on if debugging is turned on. When debugging is turned on I don't want the debug events forwarded but I do want the normal events forwarded as normal.

I have a regular expression that will only include my normal events that looks like this: [0-9]*:.*[%].*
I know that I can create a transforms.conf file in $SPLUNK_HOME/etc/apps/appName/local to filter events.

In inputs.conf I have the following:

[monitor:///var/log/boot.log]
disabled = false
followTail = 0
index = zod-os
sourcetype = linux_bootlog

I think if I add the following to transforms.conf it will do what I want:
[linux_bootlog]
REGEX = [0-9]*:.*[%].*

What I'm not 100% sure of is if I need to create a props.conf file to point to the transform like I've seen in other answers. I don't want to extract any additional fields other than what Splunk appears to automatically be doing. Also, the debug events are multiline but since they don't match the regex I think they will drop automatically.

Does all of that sound like it will work?

0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

Yes, props.conf is what tells the system when to run the transforms in transforms.conf, so there needs to be a stanza there for your source (or however you want to identify the stuff that you are treating this way).

What I would tend to do is route everything to the nullqueue and then route the matching events back.

props.conf

 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX=.* 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]*:.*[%].*
DEST_KEY = queue
FORMAT = nullQueue

The above is barely changed at all (only the second regex) from this answer... https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

Yes, props.conf is what tells the system when to run the transforms in transforms.conf, so there needs to be a stanza there for your source (or however you want to identify the stuff that you are treating this way).

What I would tend to do is route everything to the nullqueue and then route the matching events back.

props.conf

 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
 TRANSFORMS-set = setnull, setparsing

transforms.conf

[setnull]
REGEX=.* 
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [0-9]*:.*[%].*
DEST_KEY = queue
FORMAT = nullQueue

The above is barely changed at all (only the second regex) from this answer... https://answers.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk.html

morethanyell
Builder

setparsing stanza should be FORMAT = indexQueue

JordanPeterson
Path Finder

This is exactly the kind of thing I needed. Thank you very much.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...