Alerting

How to blackout the alerts

srisplunk12
Engager

Is there a way to blackout all the alerts in Splunk during the maintenance window all at one go in Splunk Cloud

Tags (2)
0 Karma

DalJeanis
Legend

You could create a blackout macro and add it once to every alert search code.

woodcock has posted a version of the code in several places -- (search for blackout and cron together and you'll find a bunch of them__ here's a few links...

His version intentionally generates errors when the search runs during a blackout, but I modified it to a version that just quietly returns no results here...

The basic method for this silenced version is to add a subsearch that figures out whether you are running in a blackout window, and if so, adds a ridiculous search term that will never be found. That makes it certain that there will be zero records returned, and it should be pretty quick in doing so.

 YOUR SEARCH HERE 
  [ | noop | stats count AS search 
    | eval search=if(((tonumber(strftime(now(),"%W"))%2)==1),"SomeLongStringHereWhichWillNeverBeFoundInBloomFilters","")] 
 | YOUR PROCESSING HERE

Note the preceding example happened to be for a cron that ran once per week, but which really only wanted to run on alternate weeks. Thus, week number odd/even test was the method of deciding blackouts. Replace ((tonumber(strftime(now(),"%W"))%2)==1) with a test that returns true() during your blackouts and false() at all other times.


You may need to develop a similar but inverse method for any alert which is designed to complain when there are zero records returned...such as for when a host stops reporting. The format of that will depend on the typical architecture of your alerts. If you need that, then please open a new question, post an example of your alert that you need to hush during blackouts, and the community will be able to help you with a version that does what you need.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...