We have a lot of scheduled search based alerts (mostly 10 minute schedules)....how do we ensure these jobs are completed before the next alert kicks in... or should we actually really care about that at all...as the crons are based on system time ?
Conversely how do we ensure that we never see alerts of a subsequent schedule before the earlier one...(this may not be relevant but just coming from a rdbms world and snapshots ) im sure Splunk is entirely different being column based ? map reduce
How do we optimize these threads ....appreciate inputs ! and everyone have a great year ahead
There is a property "realtime_schedule" in the savedsearches.conf (can't be set from Splunk Web UI), which defaults to 1. When this property value is 1, Splunk decides the execution time of a savedsearch based on its current time, hence if previous instance is already running it will skip the current instance. When this property value is 0, the Splunk decides its execution time of saved search based on its last execution time, So it will never skip an execution, but there could be delays (call continuous scheduling).
In your case I suggest to set this field to 0, so that there are never 2 instance of a search running at a time.
Ohh this has to be configured for every search level, so you're good.
Is this attribute global or per saved seatch as we have a lot of saved searches scheduled and only some of them are alerting...so we would not want a global across
Thanks
It would, but ideally the savedsearch you created should complete within the next schedule, unless there is any issue(else you should consider changing the schedule of your search if its takes more time to execute often). You can use following search to see if there are any delays in search execution. (not fully tested)
index=_internal sourcetype=\"scheduler\" |eval timediff=_time-scheduled_time | eval scheduledTime=strftime(scheduled_time,"%m/%d/%y %H:%M:%S.%3Q") | table timediff,_time, scheduledTime, status, savedsearch_name | where timediff > 60
Would this setting of "0" not lead to a cascade backlog or delayed alerts if one schedule slips....is there some way we can alert if the schedules start slipping ?
Thanks