Splunk Search

How can I extract all fields from a log file including json definiton?

florianduhme
Path Finder

I have a log file which entries/lines look like this:

12:17:35.4641 Info {"message":"TestKevin execution ended","level":"Information","logType":"Default","timeStamp":"2018-09-19T12:17:35.4641435+00:00","fingerprint":"2ee56795-e30b-4c98-b6cc-166249d18375","windowsIdentity":"RPA1\\Robotics","machineName":"RPA1","processName":"TestKevin","processVersion":"1.0.6828.23354","fileName":"Main_Alf_Production","jobId":"d415160d-8bfd-4374-b0c4-b03a35316b79","robotName":"ROBOTICS","totalExecutionTimeInSeconds":34,"totalExecutionTime":"00:00:34"}

First of all, I want to extract the time(12:17:35.4641), the Status (Info), and all of the key-value pairs included in the json definition. It would be great to do this at indexing time, so I don't have to include this logic within my search. Any Ideas?

0 Karma

joebisesi
Path Finder

So is your example the raw data?

0 Karma

florianduhme
Path Finder

Exactly, the example data above would be the _raw data.

But what I could do is to split up the fields "Time" "Status" and the JSON definition into 3 different fields (as explained in the comment to your first answer), if that would help. So we would just need to split up the key-value pairs of the field with the JSON definition.

0 Karma

joebisesi
Path Finder

If the data is all JSON, then you need this in your props.conf: KV_MODE=json

http://docs.splunk.com/Documentation/Splunk/7.1.3/Admin/Propsconf

0 Karma

florianduhme
Path Finder

I guess that's exactly my problem, that not all of my data is JSON. The first two fields are not part of the JSON definition and therefore this option does not seem to have any impact on the indexed data.

My workaround is to write the JSON to a single field (so I have 3 fields: "Time", "Status", "JSON definition") and the use the "spath" command to expand the Key-Value pairs.

index=<someindex> | spath input=jsonData 

jsonData is the field with the complete JSON definition.

But it would be nice to do this at indexing time. Any other suggestions?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...