Splunk Search

How can I create an alert that will detect a behavior change in count of events that persist for a certain amount of time?

jedatt01
Builder

I'm trying to create an alert that will trigger when the count of events is changed drastically from one time bucket to another and that change persists for a certain amount of time....say 30 minutes. I have been able to detect a change using streamstats, but I'm getting stuck on how to detect when the change persists for a certain amount of time.

The following command works to detect the change...

streamstats window=2 global=f current=t first(count) As p_count

But how do I expand that to detect a persistent change over more than two time buckets?

example:

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 90
9:15 AM MY_EVENT 80
9:20 AM MY_EVENT 85
9:25 AM MY_EVENT 75
9:30 AM MY_EVENT 95 <------------ I want alert to trigger here
9:35 AM MY_EVENT 74
9:40 AM MY_EVENT 0 

But I don't want to trigger an event if I see the following..

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 80
9:15 AM MY_EVENT 0
9:20 AM MY_EVENT 0
9:25 AM MY_EVENT 0
9:30 AM MY_EVENT 0 
9:35 AM MY_EVENT 0
9:40 AM MY_EVENT 0
0 Karma

sundareshr
Legend

Would something like this work?

... | table time event count | timechart span=30min count sum(eval(if(count>0, 1. 0))) as persist by event | where count=persist
0 Karma

jedatt01
Builder

I get where you are coming from with this, but my example table was probably too simplistic. There is a possibility that the normal value of MY_EVENT is not 0. I need a way to detect a consecutive series of abnormal values for count.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...