Alerting

How do I create an alert that only fires when threshold is exceeded and resets once results are within that threshold?

pwilly
Explorer

I have a scenario where I need to build an alert for a search that triggers on a numerical value. I need to set a threshold and alert that triggers only when that threshold is exceeded, and then have the alert reset once the value is under that threshold.

So basically, var2 returns a NUMERICAL_VALUE, which is the numerical value that I need to alert on.

For example:

  • when NUMERICAL_VALUE goes above 5, I want the alert to fire. So, if the NUMERICAL_VALUE goes from 4 to 6, the alert would fire, but not if it goes from 6 to 7.
  • If the NUMERICAL_VALUE goes from 7 back to 4, that would reset the alert. Then, if NUMERICAL_VALUE went back from 4 to 6, it would trigger the alert again.

I have the alert search set to run every two minutes.

Thanks if anyone has any answers.

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | where NUMERICAL_VALUE  > 5
Tags (2)
0 Karma

woodcock
Esteemed Legend

Keep track of your alert state in a lookup table; you need 2 searches like these:

Set Alert (fire once):

index=someindex sourcetype="somesourcetype" source="somesource" VAR1="StaticValue"  VAR2="StaticValue"
| dedup VAR1 VAR2
| where NUMERICAL_VALUE  > 5
| table VAR1 VAR2 NUMERICAL_VALUE
| lookup YourLookupFileNameHere.csv VAR1 VAR2 OUTPUT fired_time
| where isnull(fired_time)
| eval fired_time=now()
| outputlookup  coverride_if_empty=false YourLookupFileNameHere.csv

Then create another scheduled search to clear out the lookup file when NUMERICAL_VALUE < 5.

baldwintm
Path Finder

I would use a lookup table to keep the current state

index=someindex sourcetype="somesourcetype" source="somesource" Var1="StaticValue"  Var2=StaticValue | dedup VAR1 VAR2 | eval state=if(NUMERICAL_VALUE  > 5, 1, 0) | lookup alertstate.csv host OUTPUT state AS previous_state | outputlookup override_if_empty=false alertstate.csv | where previous_state<state
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...