I am using the REST API to get a large sample of JSON data every minute from the Bittrex Exchange but I would like to get each of the nested objects as their own event.
A sample of the data.
{
"success": true,
"message": "",
"result": [
{
"MarketName": "BTC-1ST",
"High": 0.00004356,
"Low": 0.00003995,
"Volume": 275838.36163835,
"Last": 0.00004,
"BaseVolume": 11.26409631,
"TimeStamp": "2018-02-22T14:26:44.74",
"Bid": 0.0000399,
"Ask": 0.0000404,
"OpenBuyOrders": 189,
"OpenSellOrders": 4095,
"PrevDay": 0.00004236,
"Created": "2017-06-06T01:22:35.727"
},
{
"MarketName": "BTC-2GIVE",
"High": 0.00000118,
"Low": 0.00000113,
"Volume": 845591.96670095,
"Last": 0.00000114,
"BaseVolume": 0.96678559,
"TimeStamp": "2018-02-22T14:26:43.647",
"Bid": 0.00000114,
"Ask": 0.00000115,
"OpenBuyOrders": 195,
"OpenSellOrders": 1235,
"PrevDay": 0.00000118,
"Created": "2016-05-16T06:44:15.287"
},...
]
}
I also have a props.conf
[bittrex-json]
TRANSFORMS-nullJsonNestingStart= removeNestingStart
TRANSFORMS-nullJsonNestingEnd= removeNestingEnd
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%f
category = Application
disabled = false
pulldown_type = true
and a the transforms.conf
[removeNestingStart]
REGEX = (\{\"success\":true,\"message\":\"\",\"result\":\[)
DEST_KEY = queue
FORMAT = nullQueue
[removeNestingEnd]
REGEX = (\]\})
DEST_KEY=queue
FORMAT = nullQueue
So I tested the regex and it matches only the top level before nested objects start. It also only catches the final "] }".
If my understanding is correct, this set up should keep the nested portions to be indexed as the other sections are sent to nullQueue.
However, when I use my props.conf sourcetype the events preview shows 0 events. Somehow my regex is matching and sending all the data to Nullqueue.
I know that I can use the spath command to extract the objects at search time. However as each object has it's own timestamp within, I'd like each object to be it's own event.
Any help is appreciated.
... View more