Getting Data In

How to keep specific events and discard the rest in props.conf and transforms.conf?

ankithreddy777
Contributor

In splunk doc it is mentioned that**

[[[Note**: In this example, the order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.

In props.conf:

[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing

2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = indexQueue]]]

why we need to put nullqueue transform first and index queue later? Putting setnull transform first do not delete events before going to setparsing transform?

0 Karma

woodcock
Esteemed Legend

There are 2 ways to trash events at the indexer:

EITHER:
You can trash the events that match a "drop me" pattern. In this case, simply send the matching events to the nullQueue.
OR:
You can send everything to nullQueue first and then pull back the events that match a "keep me" pattern to the indexQueue.

The last queue location wins.

bhargavi
Path Finder

Hi @woodcock @somesoni2 

Could you please help me out here. I have a little different scenario here, but facing similar issue.

We are integrating the json logs via HEC into Splunk Heavy Forwarder. I have tried the below configurations.I am applying the props for the source.

In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.

The configs are like below:

PROPS.CONF --

[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs

TRANSFORMS.CONF --

[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue

[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000

[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000

[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The transform on the right side is executed first, so if nullQueue is on right side (executed first), it will dump all data to nullQueue as there is no specific regex.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...