Alerting

Delete triggered alert if condition no longer matched

andrew207
Path Finder

I have an alert that runs every 1 minute and triggers when latest(status) = stopped.

If the alert runs and sees latest(status) = running, I want it to delete the triggered alert if there is one.

Is there a way to do this in Splunk?

1 Solution

michael_bates_1
Path Finder

Hello Andrew,

I do not believe there is currently a simple way to achieve this solely from within Splunk itself (happy to be proven wrong though).

Options for a possible solution would include -

  • There are REST endpoints for "fired_alerts" that will list and allow DELETE operation, however the DELETE operation cannot be called from the rest search command.
    Subsequently, this would require an external script to perform the actions, and given scripted actions is deprecated, I cannot say how long it would continue to function.

  • You could look into some alternate Alert Mgmt apps (I have deployed this one in a number of places now, https://splunkbase.splunk.com/app/2665/ )

  • Other alternatives include lookups, writing events to index, etc

Again these are alternatives, not an answer to your question.

View solution in original post

woodcock
Esteemed Legend

There is a rest endpoint to do this but you are going to have to build your own modular alert action app to do this.

0 Karma

woodcock
Esteemed Legend

What do you mean by delete the triggered alert, exactly?

0 Karma

andrew207
Path Finder

I mean literally delete the triggered alert. In the UI there's a button to delete them, in the REST API there's an endpoint to delete them. I would like an option to delete them if events occur as I described in OP

0 Karma

michael_bates_1
Path Finder

Hello Andrew,

I do not believe there is currently a simple way to achieve this solely from within Splunk itself (happy to be proven wrong though).

Options for a possible solution would include -

  • There are REST endpoints for "fired_alerts" that will list and allow DELETE operation, however the DELETE operation cannot be called from the rest search command.
    Subsequently, this would require an external script to perform the actions, and given scripted actions is deprecated, I cannot say how long it would continue to function.

  • You could look into some alternate Alert Mgmt apps (I have deployed this one in a number of places now, https://splunkbase.splunk.com/app/2665/ )

  • Other alternatives include lookups, writing events to index, etc

Again these are alternatives, not an answer to your question.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...