When do scheduled alerts get skipped?

[SplunkIsLife]

I imagine there is documentation out there, but I'm really struggling to find it. I have an alert that triggers every 2 minutes, looking back between 4 minutes ago and 2 minutes ago to see if an event occurred. I want to know under what conditions the alert will get "skipped" in favor of the one run 2 minutes later. This is an issue because of the dependence on the time frame "4 minutes ago to 2 minutes ago", as if you skip it, the next alert will be looking at a different time frame where the event would not have occurred. Does this happen when the alert is delayed more than a fixed time, or does it have to do with the time frame of the search, or neither? Thanks!

[burwell]

Hi SplunkIsLife (great username, btw). If your scheduled search/alert gets skipped you would be looking at the next two minutes of time and would not be looking at those two minutes of time that were skipped.

Why would an alert get skipped?

1) Too many alerts running at the same time

2) the previous search is still running: e.g. your schedule search from -4m to -2m.. still running.

I have occasionally had issues in 6.5 on a SHC where the captain believes the previous search is still running when it is not. For me you could see that every two minutes the same job was skipped and then eventually things would sync up and the captain would believe the job had ended and run a new search.

FYI: here's a link to a previous question about finding more out about your skipped searches.
https://answers.splunk.com/answers/514181/skipped-searches-on-shc.html#answer-513329

Here's woodcock's great answer on why a search is skipped:
https://answers.splunk.com/answers/507007/why-is-my-search-skipping.html

[somesoni2]

The alerts will get skipped if your Splunk instance or user has reached it's maximum concurrent search limit. See this for more information: https://wiki.splunk.com/Community:TroubleshootingSearchQuotas. Try to use a cron schedule so that your searches are distributed evenly throughout the hour/day (other searches may need adjustment)

It can also skip if the execution time of the search exceeds search frequency. Splunk will skip the current execution and start will next execution. This behavior is managed by following attribute in savedsearches.conf. (default 1 means it'll skip)

realtime_schedule = [0|1]
* Controls the way the scheduler computes the next execution time of a
  scheduled search.
* If this value is set to 1, the scheduler bases its determination of the next
  scheduled search execution time on the current time.
* If this value is set to 0, the scheduler bases its determination of the next
  scheduled search on the last search execution time. This is called continuous
  scheduling.
  * If set to 1, the scheduler might skip some execution periods to make sure
    that the scheduler is executing the searches running over the most recent
    time range.
  * If set to 0, the scheduler never skips scheduled execution periods.
  * However, the execution
    of the saved search might fall behind depending on the scheduler's load.
    Use continuous scheduling whenever you enable the summary index option.
* The scheduler tries to execute searches that have realtime_schedule set to 1
  before it executes searches that have continuous scheduling
  (realtime_schedule = 0).
* Defaults to 1