Splunk Search

How to alert when an "error" happens once every 10 min, but not until 3 consecutive hits?

MasterOogway
Communicator

I have an "error-string" and need to alert when I find it not only in the first 10 minute check; not only in the second 10 minute check; but when I run a third consecutive check and find the "error-string" in all three.
Is this possible?

I looked at alert throttling in 4.3.x, but that won't meet my needs. There must be some complex search string that allows for a check of "error-string" every 10 minutes, but don't alert until the third straight positive event.

Thoughts?

0 Karma

yannK
Splunk Employee
Splunk Employee

you can bucket the results per time period, and search over 30 min for :

mysearchwithsearchterms | buckets _time span=10m | stats count by _time | where count > 0

and setup a conditional of alert if number of results > 2

the results will look like :

count _time

1 2012-04-26 10:00:00
32 2012-04-26 10:10:00
4 2012-04-26 10:20:00

or

count _time

2 2012-04-26 11:40:00
6 2012-04-26 11:50:00

Time periods with 0 count will not be present, therefore, is you found less than 3 results, those are not 3 time consecutive.

MickSheppard
Path Finder

Can this string only appear once for each 10 minute period, or could it appear more than once in any 10 minute period?

0 Karma

MasterOogway
Communicator

It can appear any number of times within any given 10 minute window. But it MUST happen in three consecutive 10 minute searches.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...