I've got a splunk saved search configured to run an external script when number of events > 0. Are there any limits on how long the external script can take to run before being killed by splunk? The script simply processes the search results and updates a external datastore. The script can take between 1 seconds and 6 minutes to run, depending upon the number of events and how busy the server is. I'm seeing the following messages ocasionally in the splunkd.log
01-02-2012 02:20:33.824 -0800 WARN ScriptRunner - Killing script, probably timed out, grace=5sec, script="/local/mnt/workspace/splunk/etc/apps/search/bin/runshellscript.py"
01-02-2012 02:20:33.827 -0800 ERROR script - Script execution failed for external search command 'runshellscript'
Is there really a grace limit of 5 seconds? If so, any way to increase the timeout and allow the script to complete?
Looking at savedsearches.conf, I do not see any parameters that would allow to configure a timeout period for a scripted alert. As it is, it seems that unfortunately the script execution timeout period is hard-coded at 5 minutes with a grace period of 5 seconds.
If you would like this to be configurable, I would suggest that you submit an enhancement request by opening a support case explaining your use-case and the desired added functionality.
In lieu of being able to configure the timeout value for the fired script , you could perhaps create a solution using the Splunk REST API.
Modifying Scripted Alerts URL 1 Answer