Has anyone experienced cases where Webhook alert payloads are including strange 'loadjob' calls in the result_link??

[paimonsoror]

I recently got alerted by one of my customers that they received some strange results from one of their webhook payloads. For some reason, the payload contains a TON of 'AD' related information (we use LDAP auth for Splunk). Luckily no sensitive data is exposed, but the whole thing is incredibly bizarre.

I am used to seeing a webhook payload like this:

{
    "app": "my_super_fun_app", 
    "sid": "scheduler__admin_Y2lnbmFfY2hhcmdlYmFjaw__RMD512f1091d029f2c3a_at_1509948000_79253", 
    "owner": "admin", 
    "search_name": "Usage Drift Alert", 
    "result": {
        "Primary Contact": "--", 
        "Total Used GB": "111.54", 
        "Quota Percentage": "56.53",
        "Usage Drift": "47.86", 
        "Index": "app_sharepoint"
    }, 
    "results_link": "http://obfuscated_this:8000/app/my_super_fun_app/@go?sid=scheduler__admin_Y2lnbmFfY2hhcmdlYmFjaw__RMD512f1091d029f2c3a_at_1509948000_79253"
}

Take a look at the results_link above, it follows the typical pattern of '@go?sid=[search_sid]"

However, my customer got a result like the following:

{
    "results_link": "https://OBFUSCATED.com/app/app_harmony_pvs/search?q=%7Cloadjob%20scheduler__m38437_YXBwX2hhcm1vbnlfcHZz__RMD529b0cadb7d414a12_at_1509996600_238_D644C3A5-C833-4F31-8A77-5B88CA230AB1%20%7C%20head%2011%20%7C%20tail%201&earliest=0&latest=now",
    "app": "app_harmony_pvs",
    "search_name": "Harmony Timeouts",
    "owner": "m38437",
    "result": {
        "results.n36014.attributes.codePage{}": "",
        "results.c77827.attributes.sIDHistory{}.encoding": "",
        "results.mvruss.dn": "",
        "results.c30258.attributes.objectSid{}": "",
        "results.c30789.attributes.postalCode{}": "",
        "results.c30789.attributes.msExchHideFromAddressLists{}": "",
        "results.n36014.attributes.thumbnailPhoto{}.encoding": "",
        "results.c77827.attributes.managedObjects{}": "",
        "enabled": "",
        "app": "",
        "source": "/usr/local/openresty/nginx/logs/error.log",
        "results.spshep.attributes.homeMDB{}": "",
        "status": "",
        "returnappsonly": "false",
        "results.c77827.attributes.homeDrive{}": "",
        "results.c24086.dn": "",
        "date_wday": "monday",
        "application_server": "",
        "splunk_server": "cMASKED0010",
        "G": "",
        "range": "",
        "change_type": "",
        "ignorechanges": "false",
        "vendor_product": "",
        "timestamp": "",
        "vendor": "",
        "_sourcetype": "harmony:openresty:error",
        "date_zone": "local",
        "date_year": "2017",
        "DC": "",
        "_eventtype_color": "none",
        "Endpoint": "/CI/Relationship/iquote",
        "status_type": "",
        "results.cmedwa.attributes.manager{}": "",
        "results.c77827.attributes.msExchVersion{}": "",
        "_raw": "OBFUSCATED",
        "results.c30258.attributes.initials{}": "",
        "eventtype": [
            "err0r",
            "nix-all-logs",
            "nix_errors"
        ],
        "_indextime": "1509995250",
        "start_time": "",
        "status_description": "",
        "tag::app": "",
        "timestartpos": "0",
        "P": "",
        "punct": "//_::_[]_#:_*____(:___)______,_:_...,_:_,_:_\"_////",
        "_time": "1509995250",
        "_kv": "1",
        "user_type": "",
        "linecount": "1",
        "meta": "",
        "O": "",
        "timeendpos": "20",
        "unix_category": "all_hosts",
        "unix_group": "default",
        "CN": "",
        "tag": "error",
        "_cd": "106:4934192",
        "Verb": "GET",
        "object_category": "",
        "A": "",
        "tag::eventtype": "error",
        "date_hour": "14",
        "_bkt": "app_harmony~106~26CC9C87-0F68-4672-B76C-6C10DF00A4E2",
        "host": "cMASKED0047",
        "date_mday": "6",
        "o": "",
        "_serial": "1",
        "date_month": "november",
        "product": "",
        "src": "",
        "appserver_port_number": "",
        "splunk_server_group": "",
        "index": "app_harmony"
    },
    "sid": "scheduler__m38437_YXBwX2hhcm1vbnlfcHZz__RMD529b0cadb7d414a12_at_1509996600_238_D644C3A5-C833-4F31-8A77-5B88CA230AB1"
}

The bizaare thing is that the actual result set was a total of 12 results, with only a handful of extractions, however, if you focus on the 'result.results' node, it is a bunch of stuff from AD. I think it is only listing the users that are in the groups of the customer, and I removed a ton of them before posting, but hopefully you get the picture.

Also, take a look at the "results_link". it is completely different than the common sid based search, and instead is using a 'q=' with a loadjob. And then some stuff with a tail and 'head'.

I cant for the life of me figure out what is going on here.

[damien_chillet]

Regarding formatting of the result_link:

From my experience with Splunk talking to Jira, I could observe that
loadjob is used with head and tail when an alert is used in 'per result mode'.

If your search returned 3 results for example, there will be 3 alerts and the link for each will be:

head 1 tail 1 for the first
head 2 tail 1 for the second
head 3 tail 1 for the third

If the alert is configured in digest mode (that means you get one alert with all results provided by the search inside), Splunk will provide a link with "@go?sid="

[jkat54]

What search are you using to see this data?

[paimonsoror]

The search is:

index=app_harmony "*timed out*"

And set for -1h to now
Alert once per result

The strangest thing about the whole thing, is that this was working fine earlier today. Nothing changed on the search, and nothing changed in the infrastructure.

[jkat54]

One result says index is app harmony the other says index is app share point. Is someone feeding data into the wrong index by way of a bad inputs.conf? Maybe the stanza name is malformed

[paimonsoror]

The first result is a different search. I was using that one to demonstrate what we normally see for the formatting of the results_link .

[harsmarvania57]

Sorry but I misunderstood question earlier so I removed my comment