Alerting

How to set a threshold to events which are calculated in a certain timespan

theouhuios
Motivator

Hello

I am trying to setup a threshold limit to determine the number of Incidents created in an hour. And I need to set up an alert message if the number of incidents created in an hour (any timespan) cross the threshold limit set. Now, the threshold will be calculated upon average which would equal to total incidents for each 4 hour time segment over a 15 week period.

I am kinda lost on how to cal the avrg and how to set a threshold limit in splunk. Is it possible?

-theou

Tags (2)
0 Karma

lguinn2
Legend

Try this:

yoursearchheretoreturnincidents earliest=-105d@d 
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0) 
| bucket _time span=4h
| stats count  sum(recentEvent) as newCount by _time
| stats avg(count) as AverageCount sum(newCount) as CurrentCount
| where CurrentCount > AverageCount

Your alert condition should be "number of events greater than zero."

BTW, you are alerting on a number of events greater than average. That will happen about 50% of the time. Many people prefer to alert on a statistic like "exceeds the 95th percentile". This is also easy to calculate with Splunk:

yoursearchheretoreturnincidents earliest=-105d@d 
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0) 
| bucket _time span=4h
| stats count  sum(recentEvent) as newCount by _time
| stats p95(count) as P95Count sum(newCount) as CurrentCount
| where CurrentCount > P95Count
| rename CurrentCount as "Most Recent 4-hour Period" p95Count as "95th Percentile"

lguinn2
Legend

Then you should let Splunk calculate the standard deviation for you, too.

yoursearchheretoreturnincidents earliest=-105d@d
| eval recentEvent = if (_time>relative_time(now(),"-4h"),1,0)
| bucket _time span=4h
| stats count sum(recentEvent) as newCount by _time
| stats avg(count) as AverageCount stddev(count) as StdDeviation sum(newCount) as CurrentCount
| where CurrentCount > (AverageCount + StdDeviation)

for example

0 Karma

theouhuios
Motivator

Thanks a lot for the help. I will try this. I probably need to go ahead with the average as the PM wants it to cal it according to a standard deviation formula. 🙂 Thanks again for the help.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...