Splunk Search

Scheduled reports working inconsistently.

sajeeshpn
New Member

What could be the reasons why scheduled reports are not getting executed all the time ?

We get log messages like:-
06-05-2017 13:21:27.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:26:27.017 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:31:57.010 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:36:57.014 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:41:57.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times

and also logs like "The maximum number of historical concurrent system-wide searches has been reached", quite regularly.

Will that be a reason why scheduled reports are not working or gets skipped occasionally ? Or are there any system resources constraints (like CPU) which is causing this issue.

Please note: We only have few scheduled reports configured.

Tags (1)
0 Karma

woodcock
Esteemed Legend

What @martin_mueller said but focus on All time and Real-time searches. You should completely disable Real-time searches system-wide and you should change the default from All time to Last 15 minutes. Train your users, too.

martin_mueller
SplunkTrust
SplunkTrust

The dispatch warnings tell you that lots of search artifacts are sitting in your dispatch directory. Not a problem per se, but can slow things down a bit as mentioned in the event.

The maximum number of concurrent system-wide searches can indeed cause scheduled reports to skip. If there are no slots to run the report for an entire period, scheduled jobs would pile up infinitely.
Add more faster CPUs. Add more boxes. Speed up existing inefficient searches. Clean up existing but not needed searches. For the last two, focus on long-running and frequently-running searches.

sajeeshpn
New Member

How can we get to know that any scheduled reports have been "SKIPPED" ? Do we have any specific Splunk logs to denote that ?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...