Getting Data In

Set timeout for saved search run

kkos94
Explorer

Hello splunk community!

Is there any way to add a timeout to a saved search so that it can fail if it runs for too long?

In case this is not possible, is there another way for me to get notified when a search has been running for longer than it should(let's say, 1 hour).

Any ideas would be appreciated, thanks!

Tags (2)
0 Karma
1 Solution

DavidHourani
Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

kkos94
Explorer

dispatch.max_time did exactly what I needed to make it work.

Thanks a lot!

0 Karma

DavidHourani
Super Champion

You're welcome !

koshyk
Super Champion

You have few options available in savedsearches.conf like dispatch.max_time , auto_summarize.max_time as i'm not sure where the delay happens?

Alerting long runing queries/savedsearches are pretty straight forward. If you have monitoring Console, then the searches are already built in en-US/app/splunk_monitoring_console/search_usage_statistics_deployment., especially "Long-running Searches". You can configure alerting for any of those

Essentially the base query would look like..

(index=_audit search_group=dmc_group_search_head search_group=* action=search sourcetype=audittrail search_id!="rsa_*") 
| eval search_type=case(match(search_id,"^SummaryDirector_"),"summarization",match(search_id,"^((rt_)?scheduler__|alertsmanager_)"),"scheduled",match(search_id,"\\d{10}\\.\\d+(_[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})?$"),"ad hoc",true(),"other") 
| eval search=if((isnull(savedsearch_name) OR (savedsearch_name == "")),search,savedsearch_name) 
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host
| where total_run_time>3600

kkos94
Explorer

Thanks for your reply!

Turns out I could modify dispatch.max_time for a specific saved search instead of modifying it in the .conf file.

Good point on configuring an alert though. I will most definitely need it in the future.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...