Alerting

Delete triggered alert if condition no longer matched

andrew207
Path Finder

I have an alert that runs every 1 minute and triggers when latest(status) = stopped.

If the alert runs and sees latest(status) = running, I want it to delete the triggered alert if there is one.

Is there a way to do this in Splunk?

1 Solution

michael_bates_1
Path Finder

Hello Andrew,

I do not believe there is currently a simple way to achieve this solely from within Splunk itself (happy to be proven wrong though).

Options for a possible solution would include -

  • There are REST endpoints for "fired_alerts" that will list and allow DELETE operation, however the DELETE operation cannot be called from the rest search command.
    Subsequently, this would require an external script to perform the actions, and given scripted actions is deprecated, I cannot say how long it would continue to function.

  • You could look into some alternate Alert Mgmt apps (I have deployed this one in a number of places now, https://splunkbase.splunk.com/app/2665/ )

  • Other alternatives include lookups, writing events to index, etc

Again these are alternatives, not an answer to your question.

View solution in original post

woodcock
Esteemed Legend

There is a rest endpoint to do this but you are going to have to build your own modular alert action app to do this.

0 Karma

woodcock
Esteemed Legend

What do you mean by delete the triggered alert, exactly?

0 Karma

andrew207
Path Finder

I mean literally delete the triggered alert. In the UI there's a button to delete them, in the REST API there's an endpoint to delete them. I would like an option to delete them if events occur as I described in OP

0 Karma

michael_bates_1
Path Finder

Hello Andrew,

I do not believe there is currently a simple way to achieve this solely from within Splunk itself (happy to be proven wrong though).

Options for a possible solution would include -

  • There are REST endpoints for "fired_alerts" that will list and allow DELETE operation, however the DELETE operation cannot be called from the rest search command.
    Subsequently, this would require an external script to perform the actions, and given scripted actions is deprecated, I cannot say how long it would continue to function.

  • You could look into some alternate Alert Mgmt apps (I have deployed this one in a number of places now, https://splunkbase.splunk.com/app/2665/ )

  • Other alternatives include lookups, writing events to index, etc

Again these are alternatives, not an answer to your question.

Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...