All Apps and Add-ons

Problem using jsonkv with JSON in the logs

longdogz
Engager

I have log messages that are formatted in JSON and also include JSON data. I think that is messing up jsonkv, but I'm not sure how to get around it. I tried using backslashes before the curly braces within the data, but that didn't help. Basically, I'd like to be able to pipe through jsonkv and have the following from the JSON formatted log message:

...,"EventDescription":"RestInterface->parseInput() - minimized format={\"username\":\"$properties[username]\",\"creds\":\"$properties[password]\"}",...

Extracted as this:

EventDescription=RestInterface->parseInput() - minimized format={\"username\":\"$properties[username]\",\"creds\":\"$properties[password]\"}

Or something like that. Instead, I get something like this:

EventDescription=RestInterface->parseInput() - minimized format=

Tags (1)
0 Karma
1 Solution

vbumgarn
Path Finder

Check out spath, or jsonkvrecursive. Either of those actually parse the json instead of just using regexes. You'll probably need to pull the json itself into a field and work against that field. It seems that your field with escaped quotes is itself json, so you'll probably need to call something against that result.

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Spath

View solution in original post

0 Karma

longdogz
Engager

However, I just have a couple other things that would be nice to clean up. For one, I got some errors when it is run. Here are the errors:

command="jsonkvrecursive", Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 53, in <module> handle_dict(r, json.loads(json_text)) File "C:\Program Files\Splunk\Python-2.6\Lib\json\__init__.py", line 307, in loads return _default_decoder.decode(s) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 336, in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File "C:\Program Files\Splunk\Python-2.6\Lib\json\scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 183, in JSONObject value, end = iterscan(s, idx=end, context=context).next() File "C:\Program Files\Splunk\Python-2.6\Lib\json\scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 155, in JSONString return scanstring(match.string, match.end(), encoding, strict) ValueError: Unterminated string starting at: line 1 column 5326 (char 5326)
command="jsonkvrecursive", Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 51, in <module> json_text = raw[ raw.index( '{' ) : raw.rindex( '}' )+1 ] ValueError: substring not found 

I'm not sure if there's something I can do to avoid that error when logging?

The other thing is that when I run splunk on my log file piped to jsonkvrecursive and it pulls out the fields, it also, still, displays the full JSON log message. I'd actually prefer to only see the fields that are pulled out. It almost defeats the purpose of pulling the fields out to still see the pre-parsed JSON log message. Is there some setting on Splunk to get it to stop showing me that? Here's an example:

1 3/2/12
4:50:49.000 PM

{"TimeStamp":"03-02-12 4:50:49 PM","ComponentFileName":"C:\\Output.php","EventName":"DEBUG","ThrowableInformation":null,"componentConstituentType":"xxx","ClientIpAddress":"127.0.0.1","ThreadID":"7220","controlID":"cd0f3bcc-38fd-1d27-bd77-516952222c0a","EventDescription":"Total execution time: 0.4504","LineNumber":"395","componentName":"yyy","MethodName":"_display","RenderedMessage":"Total execution time: 0.4504","ClassName":"CI_Output"}

TimeStamp=03-02-12 4:50:49 PM   Options|  
RenderedMessage=Total execution time: 0.4504   Options|  
EventName=DEBUG   Options|  
ClientIpAddress=127.0.0.1   Options|  
controlID=cd0f3bcc-38fd-1d27-bd77-516952222c0a   Options

I'd rather just see the things at the bottom.

Thanks.

0 Karma

vbumgarn
Path Finder

Here's an example:

|stats count | eval _raw="lots of other junk {\"a\":\"b\",\"b\":{\"b\":\"c\"}}" | rex ".*?(?<json>{.*})" | spath input=json output=aa path=b.b
0 Karma

vbumgarn
Path Finder

Check out spath, or jsonkvrecursive. Either of those actually parse the json instead of just using regexes. You'll probably need to pull the json itself into a field and work against that field. It seems that your field with escaped quotes is itself json, so you'll probably need to call something against that result.

http://docs.splunk.com/Documentation/Splunk/4.3/SearchReference/Spath

0 Karma

longdogz
Engager

I think jsonkvrecursive is close to what I want. It manages to pull out the fields from my JSON log messages that actually have JSON in them.

0 Karma

longdogz
Engager

Actually, one thing I missed when posting this, the double quotes within the "value" have backslashes before them, for example:

"...format={\"username\":...

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...