Getting Data In

How to split json array into multiple events?

Sanjai676
Path Finder

Hi ,

I have this json data which I am unable to parse through any of the props.conf mechanisms.

{"meta": {"limit": 20, "next": "/api/v1/indicators/?username=xxxx&api_key=xxxxxxxxxxxxx&limit=20&offset=20", "offset": 0, "previous": null, "total_count": 570289}, "objects": [{"_id": "570f4f34c011fb78b52434d7", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "BR", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:08.677000", "impact": {"analyst": "Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:08.680000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "Basic_Feed", "date": "2016-04-14 04:05:08.679000", "method": "basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T19:26:46Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "177.33.224.193"}, {"_id": "570f4f62c011fb78b52435a1", "actions": [], "activity": [], "attack_type": "Unknown", "bucket_list": ["Basic", "US", "botnet", "low_confidence"], "campaign": [], "confidence": {"analyst": "Basic_Feed", "rating": "low"}, "created": "2016-04-14 04:05:54.227000", "impact": {"analyst": "_Basic_Feed", "rating": "unknown"}, "locations": [], "modified": "2016-04-14 04:05:54.229000", "objects": [], "relationships": [], "releasability": [], "schema_version": 3, "screenshots": [], "sectors": [], "source": [{"instances": [{"analyst": "_Basic_Feed", "date": "2016-04-14 04:05:54.229000", "method": "_basic", "reference": "REF: http://botscout.com/last_caught_cache.htm||Report Date:2016-04-13T21:26:52Z||Confidence: 65"}], "name": "osint"}], "status": "New", "threat_type": "Unknown", "tickets": [], "type": "IPv4 Address", "value": "104.238.191.144"}, 

The log is json like format, although events appear to be in one single line and I'm unable to break them using line breakers.

This is how my props.conf looks like after several different tries:

[sourcetype = _json]
BREAK_ONLY_BEFORE_DATE = false
BREAK_ONLY_BEFORE = (\{|\[\s+{)
MUST_BREAK_AFTER = (\}|\}\s+\])
SEDCMD-remove_header = s/(\{\s+.+?\[)//g
SEDCMD-remove_trailing_commas = s/\},/}/g
SEDCMD-remove_footer = s/\]\s+\}//g
TIME_PREFIX = \"modified\":\s+\"

Please help.

0 Karma
1 Solution

Sanjai676
Path Finder

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

View solution in original post

0 Karma

Sanjai676
Path Finder

Problem solved. I used the REST API modular app from Splunk and added a custom json array handler. Worked like a charm.!!

0 Karma

alexwade13
Engager

Hey! im having a similar issues, theres an array in my json that i want to grab and separate out as separate events. BREAK_ONLY_AFTER has been giving me some difficulties, what exactly do you mean by using a custom json array handler? I've gotten the data in via the REST API, all i need to do is parse it correctly.
EDIT:
I've found a different way, using SEDCMD to get rid of headers and footers of the object, and LINEBREAKER starting at the beginning of each event. i had an issue where LINEBREAKER wasn't working, where it was taking away everything in my parens, but i solved that by giving it just the comma inside the parens to eat, followed by the previous regex i had

0 Karma

ryanoconnor
Builder

See if you can start by validating the JSON. The following website is a good resource for that:

http://jsonlint.com/

Once you've done this, the easiest way to break up your JSON into key/value format would be using props.conf and setting KV_MODE to JSON. http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf

0 Karma

Sanjai676
Path Finder

i have checked and validated the json data. I had tried the KV_MODE setting,but didn't workout. I found a thread which is very similar to what i'm facing.
https://answers.splunk.com/answers/289520/how-to-split-a-json-array-into-multiple-events-wit.html
Although the same logic isn't working in my case.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...