Hi,
Is it possible for splunk to resolve PD alerts after it has gone below the threshold?
Regards
Silvano
Hey Silvano.
Yes there is.
You can actually trigger and resolve pagerduty's using the same alert even.
Take a look at the following example code.
*In production you would probably put this into a macro and pass the event_action as an argument...
index=_internal ERROR
| stats count as event_count
| eval dedup_key="ddddd"
| eval severity="warning"
| eval event_action=case(event_count>0,"trigger",1=1,"resolve")
| eval summary="A summary of this event"
| eval source="a.server.example.com"
| eval routing_key="SOME_ROUTING_KEY"
| table dedup_key,severity,event_action, summary, source, routing_key
Basically the fields above are the minimum for a pagerduty alert.
When there is one or more results the action will be to trigger an incident, when none it will send a resolve.The dedup key will end up being the name of the search so you don't need to specify.
*note, the stats count is in case there are no results as you need something to raise an event and send a resolve. This also means this only works for a single alert.
In order for this to work you need to use event rules in pagerduty.
Creeate a new event rule and create a minimum of two rules:
-The first will be resolve. ie if result.event_action=resolve then resolve.
-The second will be trigger. is if event_action=trigger then raise an incident.
There are other things you may want to do like repeat step 2 for each severity.
And that should get you auto resolving pagerduty's.
That was the best way i could find.If you found anything better since let me know.