Splunk Search

How can I create an alert that will detect a behavior change in count of events that persist for a certain amount of time?

jedatt01
Builder

I'm trying to create an alert that will trigger when the count of events is changed drastically from one time bucket to another and that change persists for a certain amount of time....say 30 minutes. I have been able to detect a change using streamstats, but I'm getting stuck on how to detect when the change persists for a certain amount of time.

The following command works to detect the change...

streamstats window=2 global=f current=t first(count) As p_count

But how do I expand that to detect a persistent change over more than two time buckets?

example:

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 90
9:15 AM MY_EVENT 80
9:20 AM MY_EVENT 85
9:25 AM MY_EVENT 75
9:30 AM MY_EVENT 95 <------------ I want alert to trigger here
9:35 AM MY_EVENT 74
9:40 AM MY_EVENT 0 

But I don't want to trigger an event if I see the following..

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 80
9:15 AM MY_EVENT 0
9:20 AM MY_EVENT 0
9:25 AM MY_EVENT 0
9:30 AM MY_EVENT 0 
9:35 AM MY_EVENT 0
9:40 AM MY_EVENT 0
0 Karma

sundareshr
Legend

Would something like this work?

... | table time event count | timechart span=30min count sum(eval(if(count>0, 1. 0))) as persist by event | where count=persist
0 Karma

jedatt01
Builder

I get where you are coming from with this, but my example table was probably too simplistic. There is a possibility that the normal value of MY_EVENT is not 0. I need a way to detect a consecutive series of abnormal values for count.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...