Splunk Search

How to trigger and terminate the alert

chprvn
New Member

Hi,

My question is on monitoring.
Currently, we have an alert which is triggered when certain percentage of call failure happen and says the service outage.
Requirement is to send a new alert when outage is terminated(service working fine)

Any thoughts?

i thought of creating 2 alerts
Alert 1- that run every 30 min if the percentage(25%) of calls fails.
Alert 2 - to check if alert 1 is triggered in last 30 min.

When ever alert 1 is triggered, it says "Outage is happening" and it also enables alert 2. (ENABLE only if its not enabled)
Then, If Alert 1 not triggered in last 30 min, alert 2 is triggered saying "Outage is over, Service is back!".and disable the Alert 2.

Is this even possible?

0 Karma

micahkemp
Champion

I think the better solution here is to structure your search to examine the past hour, evaluate each 30 minute bin independently to determine if it should alert, and alert if the most recent 30 minute bin is in the alert state and the prior 30 minutes was not.

niketn
Legend

@chprvn, what is the query for your alert?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

chprvn
New Member

Alert 1 Query is to check the percentage of call failures
Alert 2 query is to check if alert one is triggered

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...