I had this same problem and I have not found a good answer on the forums. I finally solved it, so for others who are struggling with how to throttle an alert until the next day, here is my solution.
First, don't tick the Throttle checkbox in Alerts. Instead, find out whether an alert has already been triggered for the day with a subsearch. Use the subsearch as a function that returns the value of the number of previously triggered alerts for the day. If you have not yet had an alert for the day, it will return 0, so if the alert conditions are true and alerts_for_the_day=0 then you fire the alert, otherwise no. Here is an example for checking for when you are over the license limit where the name of the alert is "License Limit Exceeded: Over 100% Usage":
| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"
Essentially here is the format:
---enter your basic alert conditions--- |
eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name=---enter the name of your alert here--- earliest=@d |
where alert_actions!="" |
stats count |
return($count)] |
where ---your basic alert triggers are evaluated--- and alert_count_today = 0 |
fields ---list the fields you want displayed in the event---
The key to making this work is understanding the event that is created every time a scheduled alert is run. The first part of the subsearch finds the alert events. In every alert event Splunk creates a field "alert_actions". If the alert is not triggered the value of alert_actions is set to an empty string. Hence the expression 'where alert_actions!=""' will only find events where an alert actually did get triggered. The "count" variable will be "0" when no alert has been triggered so far in the day, otherwise it will be "1". So if we return the value of the count variable and set the variable alert_count_today equal to the returned result, we now know whether an alert has been triggered or not.
In the case of the 300 guests, you just need to insert "where No_Of_Guests_Enrolled_in_a_day>=300 and alert_count_today=0" towards the end of the search string along with the initial search string, saved_search_name, and output fields.
... View more
