Splunk Search

How to alert when an "error" happens once every 10 min, but not until 3 consecutive hits?

MasterOogway
Communicator

I have an "error-string" and need to alert when I find it not only in the first 10 minute check; not only in the second 10 minute check; but when I run a third consecutive check and find the "error-string" in all three.
Is this possible?

I looked at alert throttling in 4.3.x, but that won't meet my needs. There must be some complex search string that allows for a check of "error-string" every 10 minutes, but don't alert until the third straight positive event.

Thoughts?

0 Karma

yannK
Splunk Employee
Splunk Employee

you can bucket the results per time period, and search over 30 min for :

mysearchwithsearchterms | buckets _time span=10m | stats count by _time | where count > 0

and setup a conditional of alert if number of results > 2

the results will look like :

count _time

1 2012-04-26 10:00:00
32 2012-04-26 10:10:00
4 2012-04-26 10:20:00

or

count _time

2 2012-04-26 11:40:00
6 2012-04-26 11:50:00

Time periods with 0 count will not be present, therefore, is you found less than 3 results, those are not 3 time consecutive.

MickSheppard
Path Finder

Can this string only appear once for each 10 minute period, or could it appear more than once in any 10 minute period?

0 Karma

MasterOogway
Communicator

It can appear any number of times within any given 10 minute window. But it MUST happen in three consecutive 10 minute searches.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...