Alerting

throttle alert once per day

marees123
Path Finder

Hi All,

My application is for guest enrollment. So new guest keep enrolling to this application, and we want to get alert if total enrollment reaches more than 300  in a day. And data must be pulled thru query for a given day. Only one alert should be triggered per day. Alert should be triggered next day if the count reaches more than 300

Query is

index="web" (host="web*" OR host="web1*") earliest = @d | search "Registered user tag: TAG id"| chart count(GUESTS_ENROLLED) as No_Of_Guests_Enrolled_in_a_day | where No_Of_Guests_Enrolled_in_a_day >=300

Requirement is

  1. Value should be searched from beginning of the day 12.01 AM
  2. email alert should be triggered once the count reaches 300 or more than that (between 12.00 AM - 11.59 PM)
  3. only one email alert should be triggered per day
  4. please advise what should be the search frequency.. like 5 mins.. 15 mins..., because alert should be triggered once the count reaches 300 (it may be between 12.00 AM to 11.59 PM anytime) . if i schedule the search to run at 11.59 PM everyday, but, if the count reaches to 300 by 2.00 AM, 22 hours gap will be there. so please advise the search frequency time as well. I guess i'm unable to use "Throttle" in search settings.

Please help.

Thanks.

0 Karma
1 Solution

jplumsdaine22
Influencer

I would do the following

  1. Change chart to streamstats

    index="web" (host="web*" OR host="web1*") earliest = @d "Registered user tag: TAG id"| streamstats count(GUESTS_ENROLLED) as No_Of_Guests_Enrolled_in_a_day | where No_Of_Guests_Enrolled_in_a_day=300

This way your alert can also contain info on who the 300th guest , so you can fold their towels into swans or something

  1. Trigger Condition should be Number of Results is Greater than 0

  2. Under Action Options in the Edit Alert section, set 'When triggered, execute actions" to "For each result"

  3. Tick the Throttle box

  4. In "Suppress results containing field value", enter date_wday

  5. In "Suppress triggering for" enter 24 hours

Your events should contain the date_wday field if your timestamps are correctly configured. The value of this field is the name of the day of the week. By using this throttling method, the alert will trigger a single time per day in a 24 hour period. at 12:00AM your date_wday field will change to the next day, so even if you reached 300 guests at 11:59PM the alert will fire for the next set of guests the following day

View solution in original post

fredchase
Explorer

I had this same problem and I have not found a good answer on the forums. I finally solved it, so for others who are struggling with how to throttle an alert until the next day, here is my solution.

First, don't tick the Throttle checkbox in Alerts. Instead, find out whether an alert has already been triggered for the day with a subsearch. Use the subsearch as a function that returns the value of the number of previously triggered alerts for the day. If you have not yet had an alert for the day, it will return 0, so if the alert conditions are true and alerts_for_the_day=0 then you fire the alert, otherwise no. Here is an example for checking for when you are over the license limit where the name of the alert is "License Limit Exceeded: Over 100% Usage":

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

Essentially here is the format:

 ---enter your basic alert conditions--- |

eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name=---enter the name of your alert here--- earliest=@d |

where alert_actions!="" |

stats count |

return($count)] |

where ---your basic alert triggers are evaluated--- and alert_count_today = 0 |

fields ---list the fields you want displayed in the event---

The key to making this work is understanding the event that is created every time a scheduled alert is run. The first part of the subsearch finds the alert events. In every alert event Splunk creates a field "alert_actions". If the alert is not triggered the value of alert_actions is set to an empty string. Hence the expression 'where alert_actions!=""' will only find events where an alert actually did get triggered. The "count" variable will be "0" when no alert has been triggered so far in the day, otherwise it will be "1". So if we return the value of the count variable and set the variable alert_count_today equal to the returned result, we now know whether an alert has been triggered or not.

In the case of the 300 guests, you just need to insert "where No_Of_Guests_Enrolled_in_a_day>=300 and alert_count_today=0" towards the end of the search string along with the initial search string, saved_search_name, and output fields.

bill_kirby
Explorer

Hey, fredchase - this is a cool idea. I haven't had a chance to try it out yet, but just wanted to give you props.

0 Karma

bill_kirby
Explorer

This looks like it's going to work for me. Thanks again, fredchase!

0 Karma

jplumsdaine22
Influencer

I would do the following

  1. Change chart to streamstats

    index="web" (host="web*" OR host="web1*") earliest = @d "Registered user tag: TAG id"| streamstats count(GUESTS_ENROLLED) as No_Of_Guests_Enrolled_in_a_day | where No_Of_Guests_Enrolled_in_a_day=300

This way your alert can also contain info on who the 300th guest , so you can fold their towels into swans or something

  1. Trigger Condition should be Number of Results is Greater than 0

  2. Under Action Options in the Edit Alert section, set 'When triggered, execute actions" to "For each result"

  3. Tick the Throttle box

  4. In "Suppress results containing field value", enter date_wday

  5. In "Suppress triggering for" enter 24 hours

Your events should contain the date_wday field if your timestamps are correctly configured. The value of this field is the name of the day of the week. By using this throttling method, the alert will trigger a single time per day in a 24 hour period. at 12:00AM your date_wday field will change to the next day, so even if you reached 300 guests at 11:59PM the alert will fire for the next set of guests the following day

marees123
Path Finder

thanks a lot.

its working perfect....

thanks...

0 Karma

jplumsdaine22
Influencer

No problem - any chance you can change your title to include the phrase 'throttle alert once per day' and add the tags alert and trigger?

0 Karma

marees123
Path Finder

yes. changed as suggested. thanks.

0 Karma

jplumsdaine22
Influencer

I would set the schedule for how soon you want to know about the alert. For example, you could run the search in real time.

Have a read through http://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts

Also have a look at the "Advanced conditional alert section here" http://docs.splunk.com/Documentation/Splunk/latest/Alert/Definescheduledalerts

0 Karma

marees123
Path Finder

thank you. yes getting alert...

but, one doubt...

i setup throttle value as 1 day, after triggering an email alert.

consider, if count reaches to 300 at 11.00 PM 23 dec 2015 and i got a mail alert. throttle value is 1 day, it means alert wont be triggered by next day, ie 24 dec 11.00 PM..

Again, next day, count reaches to 300 by 2.00 AM, 24 dec, will i get email alert? or due to that throttle value alert will be suppressed...

i guess, i'm not confusing you...

0 Karma

jplumsdaine22
Influencer

If you only want to be alerted once, instead of using throttling, just use where No_Of_Guests_Enrolled_in_a_day =300 . That way you only get alerted once (Im assuming that value doesn't go down during a day)

0 Karma

marees123
Path Finder

thanks for your suggestion. Throttle didnt work. it is not for a day. it is taking for 24 hrs.

yes, we can try where No_Of_Guests_Enrolled_in_a_day =300.

but drawback are

if the count increases from 300 to higher, between the search duration (15 minutes), we will not get alert.
if we use where No_Of_Guests_Enrolled_in_a_day >=300, we will get multiple alerts...

any idea...?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...