Alerting

How to debug Splunk alert scripts?

markb_1984
New Member

I have a script which takes all the 8 splunk args, extracts the csv containing the event details and then does some transformation before sending an alert to an IM.

To improve its reusability, I split out various parts of the script into separate shell scripts all located within a subdirectory of the main script splunk executes. Having completed is refactoring, I have successfully executed the main script passing in dummy (but sensible) values for all the expected args. Splunk is however is not successfully executing this main script.

I have followed most of the suggestions in the following wiki to no avail: http://wiki.splunk.com/Community:TroubleshootingAlertScripts

Increasing the logging to error in runshellscript.py does then reveal an error entry (in python.log) for my script when the alert fires. The error however simply outputs all the args that are passed down to the shell script. No other error is reported in this log.

Is there anywhere else I can look to see errors that might be emanating from splunk executing my script?

0 Karma

sjscott
Explorer

The easiest way to debug your scripts is to put a launcher in front of it. The idea is to redirect stdout and stderr to a file when the script runs. Thus allowing you to see what's going on. Below is what I use on RedHat for launching a python script.

#!/bin/bash
#
#  We need to lauch the python script from Bash so we can use the python version within the OS.  This
#  allows us to install the python modules we need from the package manager of the OS.   The python
#  that comes with Splunk most likely won't have the modules you require.  Thus we use the one from
#  the OS.
#
# Debug
# Use the following line to debug.  send stdout stderr to the file specified below
/usr/bin/python /opt/splunk/bin/scripts/<YOURSCRIPT> "$@" >>/var/tmp/file.txt 2>&1

# production
#/usr/bin/python /opt/splunk/bin/scripts/<YOURSCRIPT> "$@"

The "$@" takes the args that were passed to the launcher and re-passes them to the real script. The secondary reason I use a launcher is I don't want to use the version of Python that comes built into Splunk.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...