Query to find skipped scheduled searches not working correctly

[keerthana_k]

Hi

We have a set of scheduled searches which are used for creating summary indexes. To check the list of skipped searches we use the following query:

index=_internal sourcetype=scheduler status=skipped | timechart span="5m" count by savedsearch_name

This query seems to be working fine when Splunk is running. However, when we stop and start Splunk, the skipped searches in that interval does not show up in the chart. Please tell me if this is the expected behavior.

Thanks

Keerthana

[Roy_9]

Hello @woodcock ,
Did the above problem fixed by splunk support?

I did an uninstall of an app which caused splunk cloud to restart, after the restart i am seeing the SH health warning RED, 

Search Scheduler Searches Skipped
Root Cause(s):
The percentage of non high priority searches skipped (100%) over the last 24 hours is very high and exceeded the red thresholds (20%) on this Splunk instance. Total Searches that were part of this percentage=12. Total skipped Searches=12"

I am not sure, how to fix this warning, Could you please guide me?

Thanks

[woodcock · Accepted solution]

Those searches weren't "skipped", the service was down; "skipped" means that splunk did not enough available resources (e.g. CPU cores) to run all of the searches that were scheduled so it had to deliberately drop ("skip") some of them. Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely. You can file an Enhancement Request ticket but it will surely not get traction.