Alerting

How can I set an alert threshhold that varies based on event count?

sravani27
Path Finder

Hi, I want to customize my alert based on the number of events. For example, I have the query below which alerts when the failure rate is greater than 25% in 10 minutes, but the alert is too noisy when my number of calls is low.
My calls duration (for a duration of 10 minutes) throughout the day ranges from 5 to 4000.
I want to bin my calls with different alert threshold value, like if my total calls are between (5-10)--->60%, (10-20)-->50%, (20-30)-->35% and so on, rather than keeping a static threshold. Kindly please suggest. Thanks

index=abc sourcetype=abc:logs service_name="abc"|eval failure=if(response_time> 3,1,0)|timechart span=10m sum(success) as "failed_calls",count as "total_calls"|eval failure%=(failed_calls/total_calls)*100|where failure% >25
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Try like this (I believe there was typo in your original query where you're doing sum(success) instead of sum(failure), check)

index=abc sourcetype=abc:logs service_name="abc"|eval failure=if(response_time> 3,1,0)
|timechart span=10m sum(failure) as "failed_calls",count as "total_calls"|eval "failure%"=(failed_calls/total_calls)*100
| eval threshold=case(total_calls<10,60, total_calls<20,50, total_calls<30,35, .....add other conditions per your need...., true(),25)
|where 'failure%' >threshold

View solution in original post

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Try like this (I believe there was typo in your original query where you're doing sum(success) instead of sum(failure), check)

index=abc sourcetype=abc:logs service_name="abc"|eval failure=if(response_time> 3,1,0)
|timechart span=10m sum(failure) as "failed_calls",count as "total_calls"|eval "failure%"=(failed_calls/total_calls)*100
| eval threshold=case(total_calls<10,60, total_calls<20,50, total_calls<30,35, .....add other conditions per your need...., true(),25)
|where 'failure%' >threshold
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...