Alerting

subsearch condition which persists for x time

ericrobinson
Path Finder

Is there is a way to have a scheduled search run and only alert if a certain condition is met for a period of time?

We have a search that looks for a delay value that we are extracting with rex. We want to be alerted if this value is > 1 minute for a period of time. (e.g. if the value for this delay filed is greater > 1 minute for 10 minutes, send an alert)

Is this possible with the current monitoring and alerting capabilities? I only see number of events, hosts etc as a conditional alert.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

There is a conditional alert, but I suppose you are looking for a value such that the condition is met over a period of time, i.e., you want to refer to a history of running the search on a schedule. The easiest way to do this is to just run a search over a longer period of time and count (e.g., ... | bucket _time span=1min | ... by _time | stats count) but this can be very inefficient. A better way is to use a pair of searches, one to update a lookup table containing the history over time, and the second to report/alert on the lookup table.

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...