Hello dear splunketeers !
I am seeking some advice.
The splunk architecture I currently manage is fairly simple :
Everything is working fine, but I am trying to reduce the amount of data being indexed on our central splunk server
e.g :
here's a log sample that is sent by the universal forwarder
{
"@timestamp": "2018-03-02T17:15:14.756+00:00",
"@level": "INFO",
"@message": "foobar"
}
{
"@timestamp": "2018-03-02T17:15:15.800+00:00",
"@level": "ERROR",
"@message": "damnit !"
}
{
"@timestamp": "2018-03-02T17:15:16.136+00:00",
"@level": "INFO",
"@message": "foobar"
}
here's the final log that I would like to see indexed on the central server
{
"@timestamp": "2018-03-02T17:15:15.800+00:00",
"@level": "ERROR",
"@message": "damnit !"
}
So basically, I would like to get rid of all the JSON entries which level is set to "INFO"
Does anyone how to perform this ( simple ? ) filtering?
Thanks beforehand for any answer
You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...
So, in your case (all setting are done in your central Splunk server)
1) In props.conf, set the TRANSFORMS-null attribute:
[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons
2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue
3) Restart Splunk Enterprise.
You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...
So, in your case (all setting are done in your central Splunk server)
1) In props.conf, set the TRANSFORMS-null attribute:
[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons
2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue
3) Restart Splunk Enterprise.
I implemented the aformentionned settings in order to a have a redirect to the nullQueue for some entries and indeed it works as intended.
Thank you for advice !