Example of jsonutils add-on use in a search?

[dpadams]

The jsonutils application sounds like it may help considerably with my current project as we're POSTing a lot of JSON. (Likewise, xmlutils could help with some of my other projects.) I'm new enough to Splunk to have a basic question: How do I use jsonutils in a search? Do I need to use jsonutils to build some field extraction rules in a config file, or is it possible to use jsonutils in search statements on the fly?

I'd also be curious to figure out how to extract a JSON payload in its entirity from a POST. I've got the raw regex pattern I need but haven't figured out how to adapt it to Splunk. I've submitted a question about this already: http://answers.splunk.com/questions/10363/extracting-json-from-post-data.

Any would be very much appreciated!

Also, thanks to vbumgarner for writing and sharing these apps.

[dpadams]

Error below - I've added a line break between the lines for clarity.

Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 53, in  handle_dict(r, json.loads(json_text)) File "C:\Program Files\Splunk\Python-2.6\Lib\json\__init__.py", line 307, in loads return _default_decoder.decode(s) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 336, in raw_decode obj, end = self._scanner.iterscan(s, **kw).next() File "C:\Program Files\Splunk\Python-2.6\Lib\json\scanner.py", line 55, in iterscan rval, next_pos = action(m, context) File "C:\Program Files\Splunk\Python-2.6\Lib\json\decoder.py", line 171, in JSONObject raise ValueError(errmsg("Expecting property name", s, end)) ValueError: Expecting property name: line 1 column 1 (char 1)

Error : Traceback: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\jsonutils\bin\jsonkvrecursive.py", line 51, in  json_text = raw[ raw.index( '{' ) : raw.rindex( '}' )+1 ] ValueError: substring not found

[paranoid]

spath over the course of some minutes will take all memory and eventually splunkd segfaults. This problem is fixed in 4.3.2

[araitz]

paranoid - can you elaborate on this more? Have you opened a support ticket?

[paranoid]

be careful about spath, it leaks memory and can introduce instability in your system (crash)

[vbumgarn]

jsonkv is simply running some regular expressions against the event. jsonkvrecursive is trying to parse the json in the event using python's json parser, so will error if the json is not considered proper.

The spath command in Splunk 4.3 stays in process, so is going to perform better. Go with that, if you can.

[Samslara]

I'm attempting to use jsonutils and jsonkv works fine for me but I too am getting a similar error stack trace from jsonkvrecursive.

followup:
for me it seems to throw errors when the json body is invalid. In my case they're invalid because the end is truncated for being too long. When I filter those out it seems to work fine for the most part.

[mw]

The JSON utilities are splunk search commands, so they would be used on events, e.g.:

<your search> | jsonkv

The commands will attempt to extract the JSON key/value pairs into splunk fields for further processing. This is modeled after the existing "xmlkv" command which performs the same way.

It looks like you may have already figured out your other question.

[dpadams]

The error is too long for comments, I'll add an answer instead.

[dpadams]

No, there aren't any new fields in the field picker. Could it be because the source data is quoted? "{'foo':'bar'}"? I quoted the text to make it easier to insert the POST payload into a single line log format.

For jsonkvrecursive, there's what seems to be a traceback stack dump (or something of that sort) that I'll quote in a second comment.

[mw]

It shouldn't change your actual results. Do you see JSON keys in the field picker on the left-hand side (i.e. do you see host, source, sourcetype, favorite_food, whatever? What errors are you getting from jsonkvrecursive? I know that one requires valid JSON in particular, where jsonkv is a bit more liberal, but not recursive.

[dpadams]

Thanks very much for answering. I'm still not quite there yet. I've tried piping a search through jsonkv but don't get any visible change in my results. jsonkvrecursive throws a series of errors and gives up. Could the problem be that my data includes a JSON array, not only simple name-value pairs?

Imagine I've got an array of favorite foods. In my idea world, I'd like to be able to find events where "banana" is a favorite food and then get the top ten most commonly occurring foods in those events. Does that sound even remotely possible?

Thanks for any help.