All Apps and Add-ons

Problem with alert-triggered scripts for ServiceNow

gordo32
Communicator

I am trying to get the alert-triggered script working but having some difficulties as I keep getting exit code 1 on the scripts.
I'm not a python guy, so I'm unable to reverse-engineer the script, so hoping someone here can assist.

  1. I've installed the Splunk TA for ServiceNow, and configured the logon creds (setting logging to DEBUG)
  2. I have not configured any tables to be pulled down because I'm looking to push only.
  3. I ran the sample query from the documentation and it creates an Incident ticket in ServiceNow successfully:

| snowincident --category "Software" --contact_type "Phone"
--subcategory "Database" --short_description "CPU usage is high"
--ci_identifier "8214eb87c0a8018b7bd0919758dcc3c2" --priority 1
--splunk_url "hxxp://localhost:8000"

  1. I copied the $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/snow_incident.py script to $SPLUNK_HOME/etc/system/local/bin/scripts
  2. Have this alert setup to run every N minutes:

index=waf sourcetype=imperva_incapsula_cef sourceServiceName="www[dot]site[dot]com" | where isnull(cn1) | stats count as Timeouts | where Timeouts>50 | eval category="network" | eval contact_type="endpoint_security" | eval urgency=2 | eval impact=3 | eval short_description="Excessive timeouts (". Timeouts .") on www[dot]site[dot]com in the last hour" | table category, contact_type, short_description

  1. This returns the following when run manually (added commas to improve legibility): *category, contact_type, short_description * network, endpoint_security, Excessive timeouts (138) on www[dot]site[dot]com in the last hour

Now, the alert fires, and calls the python script, but:
a) There is never any debug output. I did a search for "eventtype=snow_*" over "All Time" and there are no results, so I must be failing long before the script gets to any significant portion
b) looking through the _internal logs (e.g. index=_internal snow) I see "runshellscript" instances execute passing the results.csv.gz
c) I get this error message:

ERROR script ... command="runshellscript", Script: /opt/splunk/bin/scripts/snow_incident.py exited with status code: 1

Other things I've tried:
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/*.py to /opt/splunk/bin/scripts
- copying $SPLUNK_HOME/etc/apps/Splunk_TA_snow/bin/script/snow_incident.py to /opt/splunk/bin/scripts
* (as an aside, it's kinda dumb to have 2 separate scripts with different content named snow_incident.py in this TA) *

No matter what I do, I get the status code:1 result.

BTW, in case it matters, I'm running Ubuntu 16.04 and Splunk Enterprise 6.6.4

Any help is appreciated...

gcato
Contributor

Recently had the same issue and this solution worked - configuring an alert under the Splunk_TA_snow app to send an alert to SNOW (Splunk v6.4.8).

N.B. For notification throttling to work I needed to use the snow_incident.py script instead of the snowincidentstream search command - which will, understandably, always alert in a saved search when search criteria matched.

Anyway, I wanted my alerts configured under their own app so tried softlinking the Splunk_TA_snow/bin directory into my app's directory and, "voila", my app's scripted SNOW alerts started working.

ln -s ~splunk/etc/apps/Splunk_TA_snow/bin ~splunk/etc/apps/<myappname>/bin

If you could be bothered you could probably isolate the necessary Splunk_TA_snow/bin files to a smaller selection and just copy (or softlink) the ones you need into your app's bin (and bin/scripts) directory, but in my case I did not have a bin directory so softlinking the whole Splunk_TA_snow bin works well for me. Also means any Splunk_TA_snow app upgrades should just work.

Hope someone finds this useful too.

0 Karma

PriyankaArivala
Engager

Facing same issue. Can you please tell how did you solve that issue?

0 Karma

gordo32
Communicator

Yes, as gstefancyk pointed out, after moving the Alert from being under the security context of the Search app over to the context of the SNOW add-on resolved the issue.

0 Karma

gstefancyk
Path Finder

Hi Gordo32,

Have you tried creating your alerts under the context of the Snow app and triggering the script from the default location? I had a similar issue trying to move the script to another location so I ended up just building my searches/alerts under the Splunk_TA_Snow app.

Hope that helps.

gordo32
Communicator

Finally found some time to test - and that solved it. Thanks.

0 Karma

santosh_sshanbh
Path Finder

Is there any way to create SNOW incidents without the use of Service NOW add-on? I want to use the REST API's exposed by SNOW to create the incident but not sure of to call them via alert action. Any comments on this topic would be of great help.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...