Alerting

Is it possible to purposely cause a scheduled search to be skipped?

ben_leung
Builder

This is for testing an alert to see when scheduled searches are skipped, causing the logs to write status=skipped instead of status=success OR status=continued.

Tags (2)
0 Karma
1 Solution

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

View solution in original post

woodcock
Esteemed Legend

You can conditionally short-circuit any scheduled search job by using addinfo and map for the blackout logic that it is not supposed to run like this (extra steps for clarity):

... | dedup status| status="skipped" | stats count AS blackout | addinfo | eval blackout= if(blackout>0,"YES","NO") | eval earliestMaybe=if((blackout=="NO"), info_min_time, now()) | map search="search earliest=$earliestMaybe$ latest=$info_max_time$ YOUR SEARCH HERE"

For the condition that it is not supposed to run (if the last event has status = "skipped"), the search will generate an error.

Here are other similar questions with answers that should work for you (this answer is derived from the first one):

http://answers.splunk.com/answering/260370/view.html

http://answers.splunk.com/answers/24824/can-i-set-a-blackout-period-for-a-scheduled-search-during-wh...

jrodman
Splunk Employee
Splunk Employee

I don't believe we have this functionality (force skip for testing).
Scheduled searches are skipped when they are still running at the end of the interval, so if you schedule a search to run every 5 minutes, for example, and arrange to have the search take longer than that, it will be skipped.

If you have a large index, it isn't too difficult to write a very slow search, doing something like retreiving a very large number of events, adding a |noop local then sorting the results repeatedly on different fields.

Another option would be to add a custom python search command that you can make arbitrarily slow.

ben_leung
Builder

Makes sense to make the search pass the internal time range. Testing this would be easy, but the actual query that I wanted to test runs every 15 minutes that completes relatively quick, within a minute. Thanks, I will complete my testing with this information provided.

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...