WebTools app - issues with POST bodies under version 1.2.6

[runner724]

Using version 1.2.6 of WebTools, I am running into into an issue where POSTs with request bodies are being rejected by the service I am calling due to invalid/mutilated JSON. When I run the exact same queries against WebTools version 1.2.2, they work fine. I see some subtle differences in the app's bin\curl.py file that I think are responsible.

Here is the high-level format of my query:

| makeresults 
| eval header="{\"Content-Type\":\"application/json\"}" 
| eval payload="(...)" 
| curl method=post ssl=true user=(...) pass=(...) uri="(...)" datafield=payload headerfield=header debug=true
(...)

I include the "debug=true" to demonstrate a slight difference between 1.2.2 and 1.2.6/.

Under 1.2.6, a payload like this...

{ "query": { "bool": { "must": [ {"term": (...)

... results in the 'curl_data_payload' output of:

{ "query": { "bool": { "must": [ {"term": (...)

However, under 1.2.6, the same payload results in the 'curl_data_payload' output of:

{ u'query': { u'bool': { u'must': [ {u'term': (...)

which makes sense when I look at the bin\curl.py file and find this line:

data = json.loads(result[options['datafield']])

I played around with bin\curl.py until I could get things to work. If I take this text:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            try:
                data = json.loads(result[options['datafield']])
            except:
                data = str(result[options['datafield']])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = data
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

... and make a few adjustments:

# STREAMING Use Case: iterate through results and run curl commands
if len(results) > 0:
    for result in results:
        # use JSON encoded header string if provided
        if 'headerfield' in options:
            headers = json.loads(result[options['headerfield']])
        else:
            headers = None

        # if data in options, set data = options['data']
        if 'data' in options:
            data = str(options['data'])

        # if datafield in options, set datafield = options['datafield']
        if 'datafield' in options:
            data = str(options['datafield'])
        else:
            data = None

        # debugging option
        if 'debug' in options:
            if options['debug'].lower() in ("yes", "true", "t", "1"):
                # for debugging we add results which show the options \
                # that were sent to the curl command
                result['curl_method'] = method
                result['curl_verifyssl'] = verifyssl
                result['curl_uri'] = uri
                result['curl_splunkauth'] = splunkauth
                if data != None:
                    result['curl_data_payload'] = result[data]
                if headers:
                    result['curl_header'] = headers

        # based on method, execute appropriate function
        if method.lower() in ("get","g"):
            Result = get(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("head","h"):
            Result = head(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("post","p"):
            Result = post(uri,sessionKey,verifyssl,headers,result[data],user,passwd,timeout)
        if method.lower() in ("put"):
            Result = put(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)
        if method.lower() in ("delete","del","d"):
            Result = delete(uri,sessionKey,verifyssl,headers,data,user,passwd,timeout)

The queries will work once I make those adjustments.

To be honest, I don't know enough about Python to explain why my adjustments fix things. I basically just looked at how 1.2.2 did things and adjusted the 1.2.6 to match.

Is anybody else running into cases where POSTs with a request body are rejected after updating WebTools to 1.2.6?

Splunk version: 7.1.1
'requests' python module version: 2.6.0
python version: 2.7.5 (default, Jun 11 2019, 14:33:56) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

[jkat54 · Accepted solution]

Someone suggested we make it json a while back.

You could remove the json.loads(...) from the data = lines and it should revert the behavior.

[runner724]

You're right, that is sufficient to fix the POST issues that I have, and is cleaner than changing three lines.

data = json.loads(result[options['datafield']])
->
data = str(result[options['datafield']])

Thanks!

[jkat54]

I converted to answer, please let us know if it was acceptable by accepting the answer or not. Cheers!