Getting Data In

Modular Inputs - How can I split the HTTP JSON response into multiple events?

manuel2202
Explorer

How can I split the HTTP JSON response into multiple events?
My REST API is returning a JSON Array and for each array element I like to create separate event in Splunk.
I tried sys.stdout.flush() but without success. How can I set event boundaries?

This is my json response

[{"sever1": true}, {"sever2": true}]

This is my run method

def run_script():


    try:
        cfg = get_config()
        conn = httplib.HTTPConnection(cfg["url"])
        conn.request("GET", "/")
        r1 = conn.getresponse()
        data1 = json.loads(r1.read())

        for d in data1:
            print d
            sys.stdout.flush()

        conn.close()
    except Exception as e:
        logger.error(e)
    logger.info("RUN HTTP DONE")
Tags (3)
0 Karma

manuel2202
Explorer

Each array element should be a separate event.
I solved this problem. I overlooked the SCHEMA config.

It's important to set the streaming mode to xml

SCHEME = """<scheme>
    <title>Hello</title>
    <description>Log Data to Splunk</description>
    <streaming_mode>xml</streaming_mode>
</scheme>
"""

# Empty introspection routine
def do_scheme():
    print SCHEME

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

And than I can post separate events to Splunk by using this code.

# prints XML stream
def post_data(time, data):
    print "<stream><event unbroken=\"1\"><time>%s</time><data>%s</data><done/></event></stream>"  % (time,data)

It's also important to set unbroken attribute and and the tag.

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I've moved your comment to an answer. Please accept your own answer here so that it will be flagged as having an accepted answer. - Thanks!

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Is your JSON data actually all on one line that you want to separate into different events? That is, is it like this?:

[{"sever1": true}, {"sever2": true}]

or this?:

[{"sever1": true}, 
{"sever2": true}]

What is your purpose in separating the values into different events? This sort of thing can be done at search time without having to do it at index time.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...