Alerting

Alert Recovery Notifications

keshengjie
Engager

I have a bunch of scheduled searches that occasionally trigger. Is there a way to have a recovery message sent if a subsequent scheduled search doesn't find any errors?

I need an easy way for our helpdesk to know if they should escalated these alerts.

Tags (1)
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi keshengjie

I recently had to create something similar, where it was needed to sent an up alert but only if there was down alert before. In my case I had a field named message which was used to determine if it is up or down - So this is what I did:

YourBaseSearch 
| eval isUp = if(message=="Normal","Yes","No") 
| chart latest(eval(if(isUp=="Yes",_time,null()))) as uptime earliest(eval(if(isUp=="No",_time,null()))) as downtime by domain 
| where (isnotnull(uptime) AND ( uptime >= downtime)) 
| rename domain AS Server 
| eval Status="Up again" 
| table Server Status

This saved search runs over a 2 minutes time range and sends out an alert email if in my case the checked domain is online again. Maybe this is not exactly what you asked for, but could give you a hint on how to create up alerts only if there was a down alert before.

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi keshengjie

I recently had to create something similar, where it was needed to sent an up alert but only if there was down alert before. In my case I had a field named message which was used to determine if it is up or down - So this is what I did:

YourBaseSearch 
| eval isUp = if(message=="Normal","Yes","No") 
| chart latest(eval(if(isUp=="Yes",_time,null()))) as uptime earliest(eval(if(isUp=="No",_time,null()))) as downtime by domain 
| where (isnotnull(uptime) AND ( uptime >= downtime)) 
| rename domain AS Server 
| eval Status="Up again" 
| table Server Status

This saved search runs over a 2 minutes time range and sends out an alert email if in my case the checked domain is online again. Maybe this is not exactly what you asked for, but could give you a hint on how to create up alerts only if there was a down alert before.

cheers, MuS

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...