Alerting

Receive Alert Email When condition met twice in a row

shangshin
Builder

Hi,
I have a saved search to run every 2 hours. When the number of events is more than one, an alert email is triggered. How can I make it to trigger alert email only when the same condition happened twice in a row?

e.g.

2 AM - A, B, C

4 AM - B, D, E

Alert email is triggered for event B at 4 AM

Thanks in advance!

Tags (2)
0 Karma

shangshin
Builder

Thank you. I think this should the solution. I will give it a try.

0 Karma

somesoni2
Revered Legend

The alert runs every 2 hr and if for two consecutive run some host has CPU> 90 then only an alert should be raised. With your condition, if within one 2hr period CPU> 90 for 2 time (2 15 min period) then the alert will go off, which is not desired, hence comparing two periods. Or other options can be tried, conforming to @linu1988 solution.

Value >90|bucket _time span=2h|stats count by _time | stats count |where count > 1

linu1988
Champion

which ever may be the case. If you see the average CPU utilization is more that 90% more than a time then how is it difficult to calculate?

Value >90|bucket _time span=15m|stats count(Value) as count by host|where count > 1

0 Karma

shangshin
Builder

Let's say the search is to scan a list of servers and it returns any server which cpu utilization higher than 90%. Therefore, the count is different for each run.

0 Karma

linu1988
Champion

what do you mean by condition? Isn't same as the count of the event more than 1 in 4AM? It's pretty simple.

0 Karma

somesoni2
Revered Legend

One dirty workaround could be to modify the search to run search for both the period ( if search runs at 10:00 AM then it should include result for 8:00 AM to 10:00 AM, again run for 6:00 AM to 8:00 AM) and compare count.
E.g.
search1 earliest=-2h@h latest=@h ..other terms | stats count as count1 | appendcols [search search2 earliest=-4h@h latest=-2h@h|..other terms | stats count as count2] | where count1>0 AND count2>0

Get Updates on the Splunk Community!

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...