Reporting

Why do we have 75,000 skipped scheduled searches over the last 7 days using Splunk 6.0.1?

dcroteau
Splunk Employee
Splunk Employee

Below are numbers from the search heads which support our Alerting process through Splunk.

So over the last 7 days, We have missed 75,000 schedule searches. I don't understand this because we have less than 250 scheduled searches. We need help to resolve this issue. Please let me know what "best practice action" I should take and what to check to mitigate this skipped searches. All of these searches seem to be running as admin in the Search App.

Each searchhead has 4 core x 6 CPU’s
XXXXXXXm2p had 72,000 skipped searches over the last 7 days
XXXXXXXw2p had 3,600 skipped searches over the last 7 days.

What does Skipped really mean?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely.

In other words, your scheduler could not run all the searches it was scheduled to run. This can be caused by scheduled searches actually running for much longer than expected, or by other searches blocking too much resources, or by any other resource hog.

View solution in original post

Jarohnimo
Builder

Most of these searches i found are system generated. Perhaps from a Enterprise app that goes haywire with the scheduling. I'm not sure why this isn't better controlled by splunk as out of the box functions are being skipped.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Skipped typically means a scheduled search did not finish before its next scheduled run should start - then that next run is skipped to avoid loading up the queue infinitely.

In other words, your scheduler could not run all the searches it was scheduled to run. This can be caused by scheduled searches actually running for much longer than expected, or by other searches blocking too much resources, or by any other resource hog.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...