Reporting

Why do we have 75,000 skipped scheduled searches over the last 7 days using Splunk 6.0.1?

dcroteau
Splunk Employee
Splunk Employee

Below are numbers from the search heads which support our Alerting process through Splunk.

So over the last 7 days, We have missed 75,000 schedule searches. I don't understand this because we have less than 250 scheduled searches. We need help to resolve this issue. Please let me know what "best practice action" I should take and what to check to mitigate this skipped searches. All of these searches seem to be running as admin in the Search App.

Each searchhead has 4 core x 6 CPU’s
XXXXXXXm2p had 72,000 skipped searches over the last 7 days
XXXXXXXw2p had 3,600 skipped searches over the last 7 days.

What does Skipped really mean?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely.

In other words, your scheduler could not run all the searches it was scheduled to run. This can be caused by scheduled searches actually running for much longer than expected, or by other searches blocking too much resources, or by any other resource hog.

View solution in original post

Jarohnimo
Builder

Most of these searches i found are system generated. Perhaps from a Enterprise app that goes haywire with the scheduling. I'm not sure why this isn't better controlled by splunk as out of the box functions are being skipped.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely.

In other words, your scheduler could not run all the searches it was scheduled to run. This can be caused by scheduled searches actually running for much longer than expected, or by other searches blocking too much resources, or by any other resource hog.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...