Splunk Search

How can I extract nested JSON at index time as their own event?

DHastie
Engager

I am using the REST API to get a large sample of JSON data every minute from the Bittrex Exchange but I would like to get each of the nested objects as their own event.
A sample of the data.

{
"success": true,
"message": "",
"result": [
{
"MarketName": "BTC-1ST",
"High": 0.00004356,
"Low": 0.00003995,
"Volume": 275838.36163835,
"Last": 0.00004,
"BaseVolume": 11.26409631,
"TimeStamp": "2018-02-22T14:26:44.74",
"Bid": 0.0000399,
"Ask": 0.0000404,
"OpenBuyOrders": 189,
"OpenSellOrders": 4095,
"PrevDay": 0.00004236,
"Created": "2017-06-06T01:22:35.727"
},
{
"MarketName": "BTC-2GIVE",
"High": 0.00000118,
"Low": 0.00000113,
"Volume": 845591.96670095,
"Last": 0.00000114,
"BaseVolume": 0.96678559,
"TimeStamp": "2018-02-22T14:26:43.647",
"Bid": 0.00000114,
"Ask": 0.00000115,
"OpenBuyOrders": 195,
"OpenSellOrders": 1235,
"PrevDay": 0.00000118,
"Created": "2016-05-16T06:44:15.287"
},...
]
}

I also have a props.conf

[bittrex-json]
TRANSFORMS-nullJsonNestingStart= removeNestingStart
TRANSFORMS-nullJsonNestingEnd= removeNestingEnd
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%f
category = Application
disabled = false
pulldown_type = true

and a the transforms.conf

[removeNestingStart]
REGEX = (\{\"success\":true,\"message\":\"\",\"result\":\[)
DEST_KEY = queue
FORMAT = nullQueue

[removeNestingEnd]
REGEX = (\]\})
DEST_KEY=queue
FORMAT = nullQueue

So I tested the regex and it matches only the top level before nested objects start. It also only catches the final "] }".
If my understanding is correct, this set up should keep the nested portions to be indexed as the other sections are sent to nullQueue.
However, when I use my props.conf sourcetype the events preview shows 0 events. Somehow my regex is matching and sending all the data to Nullqueue.

I know that I can use the spath command to extract the objects at search time. However as each object has it's own timestamp within, I'd like each object to be it's own event.
Any help is appreciated.

0 Karma

DHastie
Engager

Ok, in the question I am trying to remove the top level Json and trailing ]} so what remains is a bunch of Json objects {...} separated by commas.
The regex strings matched and worked but the nullQueue options will remove the whole event where it finds a match to the regex. So I was unintentionally removing the whole event.
What I needed to use was the SEDCMD in props.conf like so

SEDCMD-removeNestingStart = s/(\{\"success\":true,\"message\":\"\",\"result\":\[)//
SEDCMD-removeNestingEnd =s/(\]\})//

This succesfully removes only the matched string.

Hopefully this'll help any one elses mis-understanding

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...