How to set a threshold to events which are calculated in a certain timespan

[theouhuios]

Hello

I am trying to setup a threshold limit to determine the number of Incidents created in an hour. And I need to set up an alert message if the number of incidents created in an hour (any timespan) cross the threshold limit set. Now, the threshold will be calculated upon average which would equal to total incidents for each 4 hour time segment over a 15 week period.

I am kinda lost on how to cal the avrg and how to set a threshold limit in splunk. Is it possible?

-theou

[lguinn2]

Try this:

yoursearchheretoreturnincidents earliest=-105d@d 
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0) 
| bucket _time span=4h
| stats count  sum(recentEvent) as newCount by _time
| stats avg(count) as AverageCount sum(newCount) as CurrentCount
| where CurrentCount > AverageCount

Your alert condition should be "number of events greater than zero."

BTW, you are alerting on a number of events greater than average. That will happen about 50% of the time. Many people prefer to alert on a statistic like "exceeds the 95th percentile". This is also easy to calculate with Splunk:

yoursearchheretoreturnincidents earliest=-105d@d 
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0) 
| bucket _time span=4h
| stats count  sum(recentEvent) as newCount by _time
| stats p95(count) as P95Count sum(newCount) as CurrentCount
| where CurrentCount > P95Count
| rename CurrentCount as "Most Recent 4-hour Period" p95Count as "95th Percentile"

[lguinn2]

Then you should let Splunk calculate the standard deviation for you, too.

yoursearchheretoreturnincidents earliest=-105d@d
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0)
| bucket _time span=4h
| stats count sum(recentEvent) as newCount by _time
| stats avg(count) as AverageCount stddev(count) as StdDeviation sum(newCount) as CurrentCount
| where CurrentCount > (AverageCount + StdDeviation)

for example

[theouhuios]

Thanks a lot for the help. I will try this. I probably need to go ahead with the average as the PM wants it to cal it according to a standard deviation formula. 🙂 Thanks again for the help.