Alerting

How to create a search to check flatline for metrics?

sowji589
New Member

I am actually trying to trigger an alert when Splunk is not receiving the metrics. For now, I am checking if the value is 0 trigger an alert but I am not sure if I am doing it correct. Can someone help me in this regard? Thanks in advance.

Labels (1)
Tags (2)
0 Karma

yannK
Splunk Employee
Splunk Employee

If you know the exact origin of the data (by example host, sourcetype, fields), then you can have a search that look for that data over a recent timerange, (stats count) and trigger if there are no results at all.

 

But if you are actually searching dynamically over several origins  (| stats count by host). Then you need to compare to a list, or older data to notice that once origin is missing.

- It could be am hardcoded count, a lookup you maintain, it could be a meta data search, it could be a subsearch with a different timerange....

- or I could be a search that is looking back on a longer timerange, and do a ( | timechart count by host) Or ( | bucket _time span=1h | stats count by _time host), and has some logic to check if the recent intervals are zero). But this may require more longer/expensive searches each time, so it's not good for a frequent alert. 

0 Karma

gjanders
SplunkTrust
SplunkTrust

TrackMe might also work here as an app to monitor all data hosts, it includes metrics.

BrokenHosts and Metawoot! may also do metrics in the future.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...