Reporting

Scheduled Search Alert Suppression during Maintenance Window

dlazo
Engager

Hello, Is there a way to categorize or group Searches to suppress email alerts during a scheduled time period without having to go to each Search?

Tags (1)

tvaniderstine
Explorer

My solution, which works for environments up to medium size, is to create multple alerts,for example
switch alerts - all
switch alerts - ignore esxi ports
swtich alerts - ignore firewall ports
switch alerts - ignore hsm ports

Normally, 'switch alerts - all' is enabled, but when maintenance is going occur on esxi hosts, we disabled 'switch alerts - all' and enable 'switch alerts - ignore esxi ports'.

This allows a bit of control without writing a script to query your ticket tracking system, and using that data to query your cabling database, and then using that data to update a generic 'link state' alert because in my opinion, telling a NOC to ignore alerts is the worst thing you can do.

0 Karma

markbarber21
Path Finder

Splunk Cloud does not allow us to Enable/Disable an application without a support ticket. Our use case does not have a predefined window, but needs a control by a user.

We can accomplish this using a search Macro.

1) Include your macro at the end of each related search. Ex: "should_run"
2) To enable the search define the macro to be:

| noop

3) To disable the search define the macro to be something like:

some_really_invalid_key="this will never be found"

0 Karma

bandit
Motivator

The simplest out of the box way I can think of to achieve this is to group all alerts for a set of hosts, environment, or application into a single Splunk app space. This app space wouldn't have anything but the alerts defined. You would simply disable/enable that app in Splunk to turn off/on all associated alerts.

0 Karma

jrodman
Splunk Employee
Splunk Employee

You could also use the custom condition alert search and have that look at your maintenance window to control supression. This assumes you can express the window in terms of a splunk search.

0 Karma

netwrkr
Communicator

If I understand your question correctly - yes, you would have to make a one time change to any 'searches' that you don't want alerting during your maintenance window to filter first through a script. Instead of those searches/alerts going directly to email, tell them to use the script which we discussed above. HTH.

0 Karma

netwrkr
Communicator

Yeah, I don't think this would be too difficult. My thought is:

For all searches you do not want to send alerts during a given time period (maintenance, I assume), instead of having them send out email, have them run a script. That script could say check for the existence of a file in a given location. If that file exists, suppress email alerts. If that file does not exist, send email as normal.

0 Karma

dlazo
Engager

I think I follow but that means we would have to change each alert - During to change it and make it run script instead, correct?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...