The "level" field is being automatically added by splunk, how to we ask splunk to extract log level from my json message ?
Actually I think you can do this. The previous answer's explanation is valid, but using it you can do this:
props.conf:
[<sourcetype>]
SHOULD_LINEMERGE = false
REPORT-embedded_message = embedded_message
KV_MODE = json
props.conf:
[embedded_message]
SOURCE_KEY = message
REGEX = "(?<_KEY_1>[^:]+)":\s*"(?<_VAL_1>.*?)"
In action:
Thanks a lot again for a detailed answer. I will try this today . quick question - why do we have both KV_MODE = none
as well as KV_MODE = json
in the sourcetype section ?
Artifact of previous attempt. 🙂 WIll fix in answer.
The reason splunk isn't automatically extracting this out is (probably) because the message
field is JSON embedded in other JSON, as opposed to a sub-structure of the same JSON. I'm willing to bet the message
field contains escaped quotes if you view the raw event. Here's a run-anywhere example of what I mean:
| makeresults | eval _raw="{\"message\": \"{\\\"level\\\": \\\"0\\\"}\"}" | spath
Gives a sample _raw
of:
{"message": "{\"level\": \"0\"}"}
And a message
value of:
{"level": "0"}
Which you could run spath against and get the results you want:
| makeresults | eval _raw="{\"message\": \"{\\\"level\\\": \\\"0\\\"}\"}" | spath | spath input=message
If the message
field was part of the actual JSON structure (instead of JSON embedded in other JSON), _raw
would look more like this:
{"message": {"level": "0"}}
Which would parse properly into message.level
. The run-anywhere search to show this in action is:
| makeresults | eval _raw="{\"message\": {\"level\": \"0\"}}" | spath
So how do you make this "just work" without having to run a separate spath
each time you search? I don't think there is a way. Even if you knew that you only cared about the message
field and nothing else and you used SEDCMD
to rewrite _raw
when you index the data, it would still contain escaped quotes that are going to prevent JSON parsing from being successful.
Thanks for detailed explanation. But what bugs me is, the "req_id" field is also escaped within the message field, but somehow splunk has automatically extracted this out. Then why not the "level" ?
Are you able to see automatically extracted message.level
field on left hand side? If yes then it will give you value INFO
no, its not automatically extracted.
have you tried spath
?
http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/spath
i did try spath, but wouldn't it be easier if splunk extracts the field automatically rather than i write spath command to extract the field everytime i want to search.