Getting Data In

How to exclude JSON entries from the indexing?

craymore
New Member

Hello dear splunketeers !

I am seeking some advice.

The splunk architecture I currently manage is fairly simple :

  • one central splunk server
  • a few hundreds of universal splunk forwarder agents
  • each agent is monitoring JSON log files from our applications and sending them to our central splunk server

Everything is working fine, but I am trying to reduce the amount of data being indexed on our central splunk server

e.g :

  1. here's a log sample that is sent by the universal forwarder

    {
      "@timestamp": "2018-03-02T17:15:14.756+00:00",
      "@level": "INFO",
      "@message": "foobar"
    }
    {
      "@timestamp": "2018-03-02T17:15:15.800+00:00",
      "@level": "ERROR",
      "@message": "damnit !"
    }
    {
      "@timestamp": "2018-03-02T17:15:16.136+00:00",
      "@level": "INFO",
      "@message": "foobar"
    }
    
  2. here's the final log that I would like to see indexed on the central server

    {
      "@timestamp": "2018-03-02T17:15:15.800+00:00",
      "@level": "ERROR",
      "@message": "damnit !"
    }
    

    So basically, I would like to get rid of all the JSON entries which level is set to "INFO"

Does anyone how to perform this ( simple ? ) filtering?

Thanks beforehand for any answer

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

So, in your case (all setting are done in your central Splunk server)

1) In props.conf, set the TRANSFORMS-null attribute:

[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons

2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue

3) Restart Splunk Enterprise.

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You basically need to setup routing of specific events (events that contain @level as INFO) to nullQueue (drop from being indexed). The reference configuration could be found here: http://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...

So, in your case (all setting are done in your central Splunk server)

1) In props.conf, set the TRANSFORMS-null attribute:

[YourSourceTypeHere]
TRANSFORMS-null= drop_info_jsons

2) Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":

[drop_info_jsons]
REGEX = \"\@level\"\:\s*\"INFO\"
DEST_KEY = queue
FORMAT = nullQueue

3) Restart Splunk Enterprise.

0 Karma

craymore
New Member

I implemented the aformentionned settings in order to a have a redirect to the nullQueue for some entries and indeed it works as intended.

Thank you for advice !

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...