How to forward sub-json object to splunk indexer

[jeffreysun]

My application writes log data to disk file. The log data is one-line json as below. I use the splunker-forwarder to send the log to splunk indexer

{"line":{"level": "info","message": "data is correct","timestamp": "2017-08-01T11:35:30.375Z"},"source": "std"}

I want to only send the sub-json object {"level": "info","message": "data is correct","timestamp": "2017-08-01T11:35:30.375Z"} to splunk indexer, not the whole json. How should I configure splunk forwarder or splunk indexer? Is there something like the codec of logstash?

[jkat54]

Create a props.conf and put the sourcetype you are using in there like this:

[yourSourcetype]
SEDCMD-aaa=s/\{\"line\"://g     
SEDCMD-zzz=s/\,\"source\":\s\"std\"\}//g.

The props.conf settings will use perl based sed on the _raw to replace the stuff you're looking to drop with "nothing" (effectively removing it).

[s2_splunk]

To add, I have to ask: Can you change the application to remove the JSON attributes that are obviously not meaningful to you from the log message?
Just wondering, but that would obviously be the preferred solution since you are not wasting processor cycles on the Splunk side by removing it after the fact, which can be significant at high log volumes.

[jeffreysun]

Do you mean SEDCMD defined in the "props.conf" can only work in the heavy forwarder rather than the universal forwarder?

[jkat54]

SEDCMD should be used where Splunk "first sees" the data. I typically put it on the forwarders (heavy, or universal) and also on the indexers. In fact most people put the same props everywhere, indexer, forwarder, search head, etc. just because Splunk will know what to apply at search versus what to apply at index time. Plus some users like to upload sample data via the ui and you'll need the props on whatever server they upload to.