Splunk Search

How can I extract nested JSON at index time as their own event?

DHastie
Engager

I am using the REST API to get a large sample of JSON data every minute from the Bittrex Exchange but I would like to get each of the nested objects as their own event.
A sample of the data.

{
"success": true,
"message": "",
"result": [
{
"MarketName": "BTC-1ST",
"High": 0.00004356,
"Low": 0.00003995,
"Volume": 275838.36163835,
"Last": 0.00004,
"BaseVolume": 11.26409631,
"TimeStamp": "2018-02-22T14:26:44.74",
"Bid": 0.0000399,
"Ask": 0.0000404,
"OpenBuyOrders": 189,
"OpenSellOrders": 4095,
"PrevDay": 0.00004236,
"Created": "2017-06-06T01:22:35.727"
},
{
"MarketName": "BTC-2GIVE",
"High": 0.00000118,
"Low": 0.00000113,
"Volume": 845591.96670095,
"Last": 0.00000114,
"BaseVolume": 0.96678559,
"TimeStamp": "2018-02-22T14:26:43.647",
"Bid": 0.00000114,
"Ask": 0.00000115,
"OpenBuyOrders": 195,
"OpenSellOrders": 1235,
"PrevDay": 0.00000118,
"Created": "2016-05-16T06:44:15.287"
},...
]
}

I also have a props.conf

[bittrex-json]
TRANSFORMS-nullJsonNestingStart= removeNestingStart
TRANSFORMS-nullJsonNestingEnd= removeNestingEnd
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%f
category = Application
disabled = false
pulldown_type = true

and a the transforms.conf

[removeNestingStart]
REGEX = (\{\"success\":true,\"message\":\"\",\"result\":\[)
DEST_KEY = queue
FORMAT = nullQueue

[removeNestingEnd]
REGEX = (\]\})
DEST_KEY=queue
FORMAT = nullQueue

So I tested the regex and it matches only the top level before nested objects start. It also only catches the final "] }".
If my understanding is correct, this set up should keep the nested portions to be indexed as the other sections are sent to nullQueue.
However, when I use my props.conf sourcetype the events preview shows 0 events. Somehow my regex is matching and sending all the data to Nullqueue.

I know that I can use the spath command to extract the objects at search time. However as each object has it's own timestamp within, I'd like each object to be it's own event.
Any help is appreciated.

0 Karma

DHastie
Engager

Ok, in the question I am trying to remove the top level Json and trailing ]} so what remains is a bunch of Json objects {...} separated by commas.
The regex strings matched and worked but the nullQueue options will remove the whole event where it finds a match to the regex. So I was unintentionally removing the whole event.
What I needed to use was the SEDCMD in props.conf like so

SEDCMD-removeNestingStart = s/(\{\"success\":true,\"message\":\"\",\"result\":\[)//
SEDCMD-removeNestingEnd =s/(\]\})//

This succesfully removes only the matched string.

Hopefully this'll help any one elses mis-understanding

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...