Getting Data In

How to parse a JSON from REST API?

claudio_palmeri
Explorer

Hi all,
I need some help parsing a JSON containing none/one/multiple nested messages that I have imported via REST API (poll). I am saying one or multiple or none as it depends on what the poll is retrieving from the REST API.

In the event that the poll is retrieving no new events, I would like Splunk not to show an empty entry (square brackets only) in the log. In the event of a single message, I'd like to remove the header/footer (square brackets) and display the event. In the event of multiple messages, I would like Splunk to remove header/footer as well as split the message in individual events, with associated timestamp.

I have been doing a lot of research, however I am struggling with the various attributes in $SPLUNK_HOME/etc/system/local/props.conf and with the related regular expressions to make the whole thing work.

A sample JSON with three messages is below.

[
    {
        "source": "APPNAME",
        "id": "1234567890",
        "recorded": "2018-06-07T00:44:22.584Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email1@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message1"
        }
    },
    {
        "source": "APPNAME",
        "id": "2345678901",
        "recorded": "2018-05-07T05:12:47.409Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email2@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "POLICY",
            "message": "Details:\nIP Address: 222.222.222.222\nCountry: AU\nNew Device: true \"Default Action\"\n"
        }
    },
    {
        "source": "APPNAME",
        "id": "3456789012",
        "recorded": "2018-05-07T05:12:58.137Z",
        "action": null,
        "actors": [
            {
                "type": "user",
                "name": "email3@domain.com",
                "id": null
            }
        ],
        "resources": [],
        "client": null,
        "result": {
            "status": "SUCCESS",
            "message": "message3"
        }
    }   
]

I am not even going to bother showing what I have put in the props.conf file as it might be misleading.

I am facing this problem only when I am polling the JSON via REST API. In comparison, if I get the same JSON (flat file) and manually import it, Splunk is smart enough to break it down cleanly in individual events.

Any assistance would be highly appreciated. Thank you.

0 Karma
1 Solution

claudio_palmeri
Explorer

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

View solution in original post

0 Karma

claudio_palmeri
Explorer

Quick update, everything was resolved by using the out of the box _json sourcetype. Not sure why I was complicating my life...

0 Karma

pruthvikrishnap
Contributor

Hi,

This can be handled by creating a custome responsehandler i.e by editing responsehandler.py (/rest/bin/responsehandlers.py)

or you can exclude null values by adding below to your search.

search | where isnull(field name)

Let me know if this helps.

0 Karma

claudio_palmeri
Explorer

Hi pruthvikrishnapolavarapu,
It does help, but I wouldn't know where to start to be honest. Is this something that you can guide me through or perhaps can you recommend a Knowledge Article to follow?

Should I use the responsehandler.py would I also need to modify props.conf at all? At the moment I have a combination of SEDCMD-remove_header, SEDCMD-remove_footer, LINE_BREAKER and TIME_PREFIX that are partially doing the job, but the REGEX need some fixing...

0 Karma

claudio_palmeri
Explorer
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...