Alerting

Is it possible to purposely cause a scheduled search to be skipped?

ben_leung
Builder

This is for testing an alert to see when scheduled searches are skipped, causing the logs to write status=skipped instead of status=success OR status=continued.

Tags (2)
0 Karma
1 Solution

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

View solution in original post

woodcock
Esteemed Legend

You can conditionally short-circuit any scheduled search job by using addinfo and map for the blackout logic that it is not supposed to run like this (extra steps for clarity):

... | dedup status| status="skipped" | stats count AS blackout | addinfo | eval blackout= if(blackout>0,"YES","NO") | eval earliestMaybe=if((blackout=="NO"), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE"

For the condition that it is not supposed to run (if the last event has status = "skipped"), the search will generate an error.

Here are other similar questions with answers that should work for you (this answer is derived from the first one):

http://answers.splunk.com/answering/260370/view.html

http://answers.splunk.com/answers/24824/can-i-set-a-blackout-period-for-a-scheduled-search-during-wh...

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

ben_leung
Builder

Makes sense to make the search pass the internal time range. Testing this would be easy, but the actual query that I wanted to test runs every 15 minutes that completes relatively quick, within a minute. Thanks, I will complete my testing with this information provided.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...