I have configured a scheduled search, which runs and triggers an alert if the search returns >0 results. This search is scheduled to run every 30 minutes. When the alert is triggered, it sends an email and runs a script.
Once the alert is triggered, I want it to be throttled for the rest of the day. So the next day, from 12:00 AM on-wards, the throttle condition needs to be reset and the alert should be again triggered if and when the alert condition is met.
To implement this, I have set
alert.suppress = 1
alert.suppress.period = 1d
My understanding is, that '1d' should throttle the alert for the remaining of that day. However, I often find that the scheduled search returns a result, say at 4PM, while my alert is only triggered during a later search in the day (say 6:30PM). Is this due to the throttling that may have been introduced during the previous day's triggered alert?
When I check scheduler.log, i see the following entry:
sid="scheduler__xxx__search__RMD5623ae5e44f926665_at_1435674600_314", suppressed=1, thread_id="AlertNotifierWorker-0" 06-30-2015 07:45:01.593 -0700 INFO SavedSplunker - savedsearch_id="nobody;search;<SearchName>", user="xxx", app="search", savedsearch_name="<Search Name>", status=success, digest_mode=1, scheduled_time=1435675500, dispatch_time=1435675501, run_time=0.250, result_count=1, alert_actions=""
I see suppressed=1 even when result_count=1 during the first scheduled search of the day.
I know that the scheduled search itself doesn't have any errors, since the alert is triggered (email sent and script run) as expected, at the right time, on some days. Am I understanding it wrong? How do I ensure that the alert is always triggered the first time the scheduled search returns a result, on any day?
