Getting Data In

Modular Inputs - How can I split the HTTP JSON response into multiple events?

manuel2202
Explorer

How can I split the HTTP JSON response into multiple events?
My REST API is returning a JSON Array and for each array element I like to create separate event in Splunk.
I tried sys.stdout.flush() but without success. How can I set event boundaries?

This is my json response

[{"sever1": true}, {"sever2": true}]

This is my run method

def run_script():


    try:
        cfg = get_config()
        conn = httplib.HTTPConnection(cfg["url"])
        conn.request("GET", "/")
        r1 = conn.getresponse()
        data1 = json.loads(r1.read())

        for d in data1:
            print d
            sys.stdout.flush()

        conn.close()
    except Exception as e:
        logger.error(e)
    logger.info("RUN HTTP DONE")
Tags (3)
0 Karma

manuel2202
Explorer

Each array element should be a separate event.
I solved this problem. I overlooked the SCHEMA config.

It's important to set the streaming mode to xml

SCHEME = """<scheme>
    <title>Hello</title>
    <description>Log Data to Splunk</description>
    <streaming_mode>xml</streaming_mode>
</scheme>
"""

# Empty introspection routine
def do_scheme():
    print SCHEME

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

And than I can post separate events to Splunk by using this code.

# prints XML stream
def post_data(time, data):
    print "<stream><event unbroken=\"1\"><time>%s</time><data>%s</data><done/></event></stream>"  % (time,data)

It's also important to set unbroken attribute and and the tag.

See Documentation:
https://docs.splunk.com/Documentation/Splunk/6.6.2/AdvancedDev/ModInputsStream

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

I've moved your comment to an answer. Please accept your own answer here so that it will be flagged as having an accepted answer. - Thanks!

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Is your JSON data actually all on one line that you want to separate into different events? That is, is it like this?:

[{"sever1": true}, {"sever2": true}]

or this?:

[{"sever1": true}, 
{"sever2": true}]

What is your purpose in separating the values into different events? This sort of thing can be done at search time without having to do it at index time.

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...