Alerting

how to generate alert based on the count of unique filed value per minute over 5 minutes

Kwip
Contributor

I want to generate an alert when unique field value count is above 10 per minute for 5 minutes.

Example:
my search giving me 3 values in host field.
host name---Count
server 1 ---- 10
server 2 ---- 26
Server 3 ---- 8

I want to generate alert if any of these servers (must be same all over the 5 minutes ) remains more than 10 per 1 minute over continues 5 mins.
Like
Time-----------Host Name-- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server1 ---- 15
03/02 07:24 - server1 ---- 16
03/02 07:25 - server1 ---- 22

I do not want to generate alert if combination of servers remains more than 10 per minute for continues 5 mins.
Like
Time---------- Host Name -- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server2 ---- 15 (During this time server1 count is less than 10. ie, server1 5)
03/02 07:24 - server3 ---- 16 (During this time server1 count is less than 10. ie, server1 9)
03/02 07:25 - server1 ---- 22

1 Solution

DalJeanis
SplunkTrust
SplunkTrust
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

Kwip
Contributor

@DalJeanis
The above query did the magic!!! Thank you very much!!!

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Quite welcome. Changed overall time frame to 6m from 7m after verifying that the alert would work correctly with only 6m. I had been wondering why 7m was needed, and found that my test bed had required 7 minutes only because of an internal "| head 1000" pipe on my base search test query that often stopped the search results early... 😉

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...