Alerting

Raise alert on status change only

atelesca
Explorer

Hello,
I would like to be able to raise an alert when the status of a condition changes?
For example:
- when the CPU user percentage is higher than 80% I would like to raise an alert that launch a command raising the alarm_on to an external program. This should be done only if the previous CPU user percentage was lower than 80%. There is no need to raise the alarm again.
- when the CPU user percentage goes back to below 80% I would like to send a command to my external program that set the alarm_off. This has to be done only if the previous CPU user percentage was higher than 80%.
Do you know how can I do it in Splunk?
I could raise an alert based on a search such as
sourcetype="cpu"| where CPU="all" and pctUser>20
and add the conditions to
sourcetype="cpu"| where CPU="all" and pctUser>20 | stats count=1
so that I am sure that the condition has been met only once.
The problem is:
- I should specify the time, like last 2 minutes
- How can I make sure that this is met by host?

Thanks for your help.
Cheers,
Adriana

Tags (1)
0 Karma

atelesca
Explorer

Well, at the end the only way I found is to create one alert that raise the alarm when the following condition is met:

sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c>80

that runs every minute and it looks at the past 2 minutes.

In order to unraise the alarm, I created another alert that is launched when the following condition is met:
sourcetype=cpu earliest=-2m | multikv | where CPU="all" | stats count(eval(pctUser > 80)) as a ,count(eval(pctUser < 80)) as b, latest(pctUser) as c, by host | search a =1 AND b = 1 AND c<80

it also runs every minute and looks at the past 2 minutes.

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...