Alerting

Why are skipped alerts appearing?

bhavneeshvohra
Engager

We have two scheduled alerts :

1) 1st fetch records for last 30 mins and cron expression :29,59 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * *

2) 2nd fetch records for last 6hrs and cron expression:59 5,11,17,23 * * * . .

Here we have found 3 Alerts for 30 mins are skipped, which should be occurred at 10:59 PM, 11:29 PM , 11:59 PM

and 1 Alert for 6 hrs skipped at 11:59 PM.

Why these alerts are getting skipped.?

Please help!!!

0 Karma

solarboyz1
Builder
index=_internal  sourcetype=scheduler  status="skipped" savedsearch_name="YOUR_SAVED_SEARCH"

You should get events like:

08-25-2019 21:30:02.860 +0000 INFO SavedSplunker - savedsearch_id="nobody;SA-ThreatIntelligence;Threat - Correlation Searches - Lookup Gen", search_type="scheduled", user="admin", app="SA-ThreatIntelligence", savedsearch_name="Threat - Correlation Searches - Lookup Gen", priority=default, status=skipped, reason="The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached", concurrency_category="historical_scheduled", concurrency_context="saved-search_cluster-wide", concurrency_limit=1, scheduled_time=1566768600, window_time=0

Which should show the reason it was skipped.

0 Karma

bhavneeshvohra
Engager

Thanks for the reply!!
"The reason we found is maximum number of concurrent search jobs has been reached".
How should w e solve this ? What should be done such that no alerts get skipped anyhow?

0 Karma

solarboyz1
Builder

If you've reached the maximum number of concurrent search jobs. You need to either increase the limits (if you've got the infrastructure to support it) or reduce the number of searches being run as @snigdhasaxena
describes

0 Karma

snigdhasaxena
Communicator

@bhavneeshvohra raise the priority of your alert, disable the jobs are that running on regular basis and no longer needed or kill the jobs that are already running that have been in queue for long.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...