Scheduled reports working inconsistently.

[sajeeshpn]

What could be the reasons why scheduled reports are not getting executed all the time ?

We get log messages like:-
06-05-2017 13:21:27.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:26:27.017 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:31:57.010 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:36:57.014 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times
06-05-2017 13:41:57.015 +0000 WARN DispatchReaper - Too many search jobs found in the dispatch directory (found=3069, warning level=3000). This could negatively impact search starting times

and also logs like "The maximum number of historical concurrent system-wide searches has been reached", quite regularly.

Will that be a reason why scheduled reports are not working or gets skipped occasionally ? Or are there any system resources constraints (like CPU) which is causing this issue.

Please note: We only have few scheduled reports configured.

[woodcock]

What @martin_mueller said but focus on All time and Real-time searches. You should completely disable Real-time searches system-wide and you should change the default from All time to Last 15 minutes. Train your users, too.

[martin_mueller]

The dispatch warnings tell you that lots of search artifacts are sitting in your dispatch directory. Not a problem per se, but can slow things down a bit as mentioned in the event.

The maximum number of concurrent system-wide searches can indeed cause scheduled reports to skip. If there are no slots to run the report for an entire period, scheduled jobs would pile up infinitely.
Add more faster CPUs. Add more boxes. Speed up existing inefficient searches. Clean up existing but not needed searches. For the last two, focus on long-running and frequently-running searches.

[sajeeshpn]

How can we get to know that any scheduled reports have been "SKIPPED" ? Do we have any specific Splunk logs to denote that ?