All Apps and Add-ons

PagerDuty autoresolve of alerts

silvanop
New Member

Hi,

Is it possible for splunk to resolve PD alerts after it has gone below the threshold?

Regards
Silvano

0 Karma

jethrop
Explorer

Hey Silvano.

 

Yes there is.

You can actually trigger and resolve pagerduty's using the same alert even.

Take a look at the following example code.

*In production you would probably put this into a macro and pass the event_action as an argument...

index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key

Basically the fields above are the minimum for a pagerduty alert.

When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.

*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.

In order for this to work you need to use event rules in pagerduty.

Creeate a new event rule and create a minimum of two rules:

-The first will be resolve. ie if result.event_action=resolve then resolve.

-The second will be trigger. is if event_action=trigger then raise an incident.

 

There are other things you may want to do like repeat step 2 for each severity.

And that should get you auto resolving pagerduty's.

That was the best way i could find.If you found anything better since let me know.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...