Alerting

subsearch condition which persists for x time

ericrobinson
Path Finder

Is there is a way to have a scheduled search run and only alert if a certain condition is met for a period of time?

We have a search that looks for a delay value that we are extracting with rex. We want to be alerted if this value is > 1 minute for a period of time. (e.g. if the value for this delay filed is greater > 1 minute for 10 minutes, send an alert)

Is this possible with the current monitoring and alerting capabilities? I only see number of events, hosts etc as a conditional alert.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

There is a conditional alert, but I suppose you are looking for a value such that the condition is met over a period of time, i.e., you want to refer to a history of running the search on a schedule. The easiest way to do this is to just run a search over a longer period of time and count (e.g., ... | bucket _time span=1min | ... by _time | stats count) but this can be very inefficient. A better way is to use a pair of searches, one to update a lookup table containing the history over time, and the second to report/alert on the lookup table.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...