Splunk Search

Ensure scheduled searched based alerts finish before next one kicks in

Mag2sub
Path Finder

We have a lot of scheduled search based alerts (mostly 10 minute schedules)....how do we ensure these jobs are completed before the next alert kicks in... or should we actually really care about that at all...as the crons are based on system time ?

Conversely how do we ensure that we never see alerts of a subsequent schedule before the earlier one...(this may not be relevant but just coming from a rdbms world and snapshots ) im sure Splunk is entirely different being column based ? map reduce

How do we optimize these threads ....appreciate inputs ! and everyone have a great year ahead

Tags (1)
0 Karma

somesoni2
SplunkTrust
SplunkTrust

There is a property "realtime_schedule" in the savedsearches.conf (can't be set from Splunk Web UI), which defaults to 1. When this property value is 1, Splunk decides the execution time of a savedsearch based on its current time, hence if previous instance is already running it will skip the current instance. When this property value is 0, the Splunk decides its execution time of saved search based on its last execution time, So it will never skip an execution, but there could be delays (call continuous scheduling).

In your case I suggest to set this field to 0, so that there are never 2 instance of a search running at a time.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ohh this has to be configured for every search level, so you're good.

0 Karma

Mag2sub
Path Finder

Is this attribute global or per saved seatch as we have a lot of saved searches scheduled and only some of them are alerting...so we would not want a global across

Thanks

0 Karma

somesoni2
SplunkTrust
SplunkTrust

It would, but ideally the savedsearch you created should complete within the next schedule, unless there is any issue(else you should consider changing the schedule of your search if its takes more time to execute often). You can use following search to see if there are any delays in search execution. (not fully tested)

index=_internal sourcetype=\"scheduler\" |eval timediff=_time-scheduled_time | eval scheduledTime=strftime(scheduled_time,"%m/%d/%y %H:%M:%S.%3Q") | table timediff,_time, scheduledTime, status, savedsearch_name | where timediff > 60

0 Karma

Mag2sub
Path Finder

Would this setting of "0" not lead to a cascade backlog or delayed alerts if one schedule slips....is there some way we can alert if the schedules start slipping ?

Thanks

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...