Splunk Search

All events except one to Null

jordanperks
Path Finder

I am getting millions of events/day that I need to send to the null queue. I need to match all events with the exception of anything that says "Windows XP" and send them to null. Any thoughts on writing this?

My initial thought would be to write something that matches Windows XP and sends it to a specific index and put that above another entry that sends everything to Null. Thoughts?

Tags (2)
0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi jordanperks,

take a look at the second example from the docs about filter events and sent to queue, basically it is the other way around. In this example you will keep only sshd events from the messages file. The order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

hope this helps ...

cheers, MuS

View solution in original post

linu1988
Champion

Hello,
You need to use the props.conf and transforms.conf. Should be put in indexer.

Props.conf:

[Source_Name]
TRANSFORMS-log=xp-redirect

Transforms.conf:

[xp-redirect]
REGEX=\s*Windows XP\s*
DEST_KEY=queue
FORMAT=nullQueue

Thanks

jordanperks
Path Finder

The REGEX was exactly what I needed, but this filtered out the XP events rather than kept them.

props.conf:

[host::myhostname]
TRANSFORMS-null = hostnull,hostparsing

Transforms.conf:

[hostnull]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

[hostparsing]
REGEX=\s*Windows XP\s*
DEST_KEY = queue
FORMAT = indexQueue

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi jordanperks,

take a look at the second example from the docs about filter events and sent to queue, basically it is the other way around. In this example you will keep only sshd events from the messages file. The order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue

hope this helps ...

cheers, MuS

bhargavi
Path Finder

Hi @MuS 

Could you please help me out here.  We are integrating the json logs via HEC into Splunk Heavy Forwarder.

I have tried the below configurations.I am applying the props for the source.

In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.

The configs are like below:

PROPS.CONF --

[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs

TRANSFORMS.CONF --

[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue

[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000

[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000

[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true

0 Karma

jordanperks
Path Finder

This worked perfectly.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...