Alerting

Is there a time limit on saved search scripts?

heybigben
Explorer

I've got a splunk saved search configured to run an external script when number of events > 0. Are there any limits on how long the external script can take to run before being killed by splunk? The script simply processes the search results and updates a external datastore. The script can take between 1 seconds and 6 minutes to run, depending upon the number of events and how busy the server is. I'm seeing the following messages ocasionally in the splunkd.log

01-02-2012 02:20:33.824 -0800 WARN ScriptRunner - Killing script, probably timed out, grace=5sec, script="/local/mnt/workspace/splunk/etc/apps/search/bin/runshellscript.py"

01-02-2012 02:20:33.827 -0800 ERROR script - Script execution failed for external search command 'runshellscript'

Is there really a grace limit of 5 seconds? If so, any way to increase the timeout and allow the script to complete?

Labels (1)
Tags (1)

chandanjaisal
Explorer

https://answers.splunk.com/answers/653458/max-time-spent-in-per-result-alerts-issue.html

limits.conf

[scheduler]
action_execution_threads = 10
actions_queue_size = 10000
max_per_result_alerts =10000
max_per_result_alerts_time = 600

nembela
Path Finder

I don't know if this problem relevant yet but I had the same problem. Splunk support found an undocumented paramater for this in savedseaches.conf:

action.script.maxtime
The maximum amount of time a script action takes before the action is canceled. The valid format is number followed by a time unit ("s", "m", "h", or "d").

I hope it helps.

Damien_Dallimor
Ultra Champion

In lieu of being able to configure the timeout value for the fired script , you could perhaps create a solution using the Splunk REST API.

ie:

  • write a python script that is cron triggered.
  • python script uses the Splunk Python SDK to execute your search
  • then proceed with your existing logic if search returns "events > 0"
0 Karma

hexx
Splunk Employee
Splunk Employee

Looking at savedsearches.conf, I do not see any parameters that would allow to configure a timeout period for a scripted alert. As it is, it seems that unfortunately the script execution timeout period is hard-coded at 5 minutes with a grace period of 5 seconds.

If you would like this to be configurable, I would suggest that you submit an enhancement request by opening a support case explaining your use-case and the desired added functionality.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...