Alerting

Alert on |search field=0 returns too many rows?

bmgilmore
Path Finder

I've got a scheduled search that calculates the variability of a numeric field over time that should always be moving. If I pipe a "search Variability=0" at the end, and run in search view, the search runs, shows a lot of rows at first as it calculates back through time, and then shows the correct number of rows.

Oddly, when I set this search to alarm, it sends many alarms and the CSV search results attached show many (almost all) of the available rows, each with a Variability of 0. Returning to the triggered job returns 0 (or whatever 1 or 2) rows as I would expect. Especially strange is that each time the alert triggers, it is a new number of false positive rows.

It's almost as if the search is not waiting to complete before triggering the alert? Any ideas here? Thanks in advance!

Tags (1)
0 Karma

jonuwz
Influencer

You probably need a custom condition in your alert

i.e.

main search

... | stats count(eval(Variability==0)) as not_variable

then a custom condition where

search not_variable > 0
0 Karma

MHibbin
Influencer

attached? ... you will need to paste it in the question as code (using the button "101010", or by starting each line of the code with 4 spaces, or by inclosing in backticks).

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...