I am getting millions of events/day that I need to send to the null queue. I need to match all events with the exception of anything that says "Windows XP" and send them to null. Any thoughts on writing this?
My initial thought would be to write something that matches Windows XP and sends it to a specific index and put that above another entry that sends everything to Null. Thoughts?
Hi jordanperks,
take a look at the second example from the docs about filter events and sent to queue, basically it is the other way around. In this example you will keep only sshd
events from the messages
file. The order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
In props.conf:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue
hope this helps ...
cheers, MuS
Hello,
You need to use the props.conf and transforms.conf. Should be put in indexer.
Props.conf:
[Source_Name]
TRANSFORMS-log=xp-redirect
Transforms.conf:
[xp-redirect]
REGEX=\s*Windows XP\s*
DEST_KEY=queue
FORMAT=nullQueue
Thanks
The REGEX was exactly what I needed, but this filtered out the XP events rather than kept them.
props.conf:
[host::myhostname]
TRANSFORMS-null = hostnull,hostparsing
Transforms.conf:
[hostnull]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue
[hostparsing]
REGEX=\s*Windows XP\s*
DEST_KEY = queue
FORMAT = indexQueue
Hi jordanperks,
take a look at the second example from the docs about filter events and sent to queue, basically it is the other way around. In this example you will keep only sshd
events from the messages
file. The order of the transforms in props.conf matters. The null queue transform must come first; if it comes later, it will invalidate the previous transform and route all events to the null queue.
In props.conf:
[source::/var/log/messages]
TRANSFORMS-set= setnull,setparsing
In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = \[sshd\]
DEST_KEY = queue
FORMAT = indexQueue
hope this helps ...
cheers, MuS
Hi @MuS
Could you please help me out here. We are integrating the json logs via HEC into Splunk Heavy Forwarder.
I have tried the below configurations.I am applying the props for the source.
In transforms, there are different regexes and I would want to route it to different indexes based on log files and route all the other files not required to a null queue. I would not be able to use FORMAT=indexqueue in transforms.conf as I cannot mention multiple indexes in inputs.conf .This is not working and no data is getting indexed. Kindly help.
The configs are like below:
PROPS.CONF --
[source::*model-app*]
TRANSFORMS-segment=setnull,security_logs,application_logs,provisioning_logs
TRANSFORMS.CONF --
[setnull]
REGEX=class\"\:\"(.*?)\"
DEST_KEY = queue
FORMAT = nullQueue
[security_logs]
REGEX=(class\"\:\"(/var/log/cron|/var/log/audit/audit.log|/var/log/messages|/var/log/secure)\")
DEST_KEY=_MetaData:Index
FORMAT=model_sec
WRITE_META=true
LOOKAHEAD=40000
[application_logs]
REGEX=(class\"\:\"(/var/log/application.log|/var/log/local*?.log)\")
DEST_KEY=_MetaData:Index
FORMAT=model_app
WRITE_META=true
LOOKAHEAD=40000
[provisioning_logs]
REGEX=class\"\:\"(/opt/provgw-error_msg.log|/opt/provgw-bulkrequest.log|/opt/provgw/provgw-spml_command.log.*?)\"
DEST_KEY=_MetaData:Index
FORMAT=model_prov
WRITE_META=true
This worked perfectly.