Alerting

How do I suppress alerts until the next day at 12 am and not 24 hours?

splunkfmpa
New Member

I have a couple of alerts for License usage set to check every hour when they exceed 75 %. At the moment, I receive an email every hour after it reaches the threshold. I only want to be notified one time for that day. If I use the throttle option and suppress for 1 day, it will wait 24 hours until it alerts me again. This is an issue because if the threshold is exceeded earlier the following day, I won't get notified until that time runs out. I would like to suppress the alert until 12 am that night. Can anyone please help? Thank you

0 Karma

fredchase
Explorer

I posted a solution to this general problem here: https://answers.splunk.com/answers/337985/throttle-alert-once-per-day.html
In short, the search string I used to trigger once a day alerting when over the license limit is this:

| rest splunk_server=local /services/licenser/pools | rename title AS Pool | search [rest splunk_server=local /services/licenser/groups | search is_active=1 | eval stack_id=stack_ids | fields stack_id] | eval quota=if(isnull(effective_quota),quota,effective_quota) | eval "Percentage of daily license limit used"=round(used_bytes/quota*100,2) | eval "Alert time"=strftime(now(), "%T %Z") | eval alert_count_today=[search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="License Limit Exceeded: Over 100% Usage" earliest=@d | where alert_actions!="" | stats count | return($count)] | where 'Percentage of daily license limit used' > 100 and alert_count_today = 0 | fields "Alert time" "Percentage of daily license limit used"

For more details see the original post.

somesoni2
SplunkTrust
SplunkTrust

Try this as your alert search (no throttling needs to be set)

| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | appendcols [search index=_internal sourcetype=scheduler thread_id=AlertNotifier* savedsearch_name="PUTYOURALERTSEARCHNAMEHERE" earliest=@d | head 1 | table _time]  | where '% used' > 75 AND isnull(_time)| fields "% used", "updated" 

The appendcols subsearch will try to find the timestamp of an alert that was fired today. If it finds one, the alert will be be fired.

splunkfmpa
New Member

Thank you, I inserted the search and will test today. I sent ahead and removed any throttle in the Trigger Condition screen.

0 Karma

fmpa_isaac
Path Finder

It was able to alert me based on the threshold I set. However, it continues to send an alert. How would you schedule this report in the Type and Trigger condition?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is your alert search ?

0 Karma

splunkfmpa
New Member
| rest /services/licenser/usage | eval "% used"=round(slaves_usage_bytes/quota*100,2) | where '% used' > 75 | fields "% used", "updated"
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...