Splunk Search

How can I create an alert that will detect a behavior change in count of events that persist for a certain amount of time?

jedatt01
Builder

I'm trying to create an alert that will trigger when the count of events is changed drastically from one time bucket to another and that change persists for a certain amount of time....say 30 minutes. I have been able to detect a change using streamstats, but I'm getting stuck on how to detect when the change persists for a certain amount of time.

The following command works to detect the change...

streamstats window=2 global=f current=t first(count) As p_count

But how do I expand that to detect a persistent change over more than two time buckets?

example:

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 90
9:15 AM MY_EVENT 80
9:20 AM MY_EVENT 85
9:25 AM MY_EVENT 75
9:30 AM MY_EVENT 95 <------------ I want alert to trigger here
9:35 AM MY_EVENT 74
9:40 AM MY_EVENT 0 

But I don't want to trigger an event if I see the following..

time        event         count
9:00 AM MY_EVENT 0
9:05 AM MY_EVENT 75
9:10 AM MY_EVENT 80
9:15 AM MY_EVENT 0
9:20 AM MY_EVENT 0
9:25 AM MY_EVENT 0
9:30 AM MY_EVENT 0 
9:35 AM MY_EVENT 0
9:40 AM MY_EVENT 0
0 Karma

sundareshr
Legend

Would something like this work?

... | table time event count | timechart span=30min count sum(eval(if(count>0, 1. 0))) as persist by event | where count=persist
0 Karma

jedatt01
Builder

I get where you are coming from with this, but my example table was probably too simplistic. There is a possibility that the normal value of MY_EVENT is not 0. I need a way to detect a consecutive series of abnormal values for count.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...