Alerting

How can I create an alert to generate and compare results on new values only?

Robbie1194
Communicator

I want to find out which UF's have stopped logging in my estate, and while I try fix these UF's, I don't want a daily alert with the same results.

I only want to see any alerts if it's different UF's that have stopped reporting, I don't want to see any old results with the new results either as it might get difficult to track what is resolved and what isn't.

The end result is that a separate alert is created every time a new forwarder stops logging. I know how to write the search to find missing forwarders, does anyone know how I should compare results to only generate on new values?

Cheers.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi robbie1194,
you can find a solution to your need in DMC.

Anyway, to do something like you want I created a lookup (called perimeter.csv) containing all the forwarders in my perimeter (lookup has only one column called "host") and I check if I receive logs from each of them.
I run this search on _internal index every 5 minutes:

index=_internal 
| eval host=upper(host)
| stats count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host), count=0 | fields host count ]
| stats sum(count) AS Total BY host
| where Total=0

In this way you have an alert that triggers all the forwarders in your perimeter that don't send logs every five minutes and you can intervene without compare today's results with yesterday's results.
You can also display the situation in a dashboard also in graphic mode.

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...