Alerting

How to get one alert for all the same alerts?

cramery
New Member

I want to make an Alert, as soon as a Application gets startet (in this case Firefox). But for somehow, always 4 or more alerts get triggered. But the Alerts are exact the same to each other, theres not a single diffrent in the whole alert. How can I reach, that I only get one Alert for all of these same Alerts, that are also in the exact same time? (Also, when I just use it as a search, it perfectly works fine too)

My Search:

source="WinEventLog:Microsoft-Windows-Sysmon/Operational" (Image="C:\\Program Files\\Mozilla Firefox\\firefox.exe") | stats first(*) by ParentProcessId

My Alert conditions:

Real-Time
Number of Results: Equals 2 (It only works with 2, dont know why)
In 1 Minute
Triggers Once
Add to Triggered Alerts
Tags (2)
0 Karma

woodcock
Esteemed Legend

First of all, stop using realtime and your problem goes away; then search like this:

source="WinEventLog:Microsoft-Windows-Sysmon/Operational" (Image="C:\\Program Files\\Mozilla Firefox\\firefox.exe") | stats first(*) by ParentProcessId host
0 Karma

nickhills
Ultra Champion

This is the problem with Real-Time alerts.

Consider this:
you have a search which generates a result (like yours)
It is looking for events which have occurred in the last 60 seconds.

Your event fires at 05 seconds past the minute - the RT search runs, it fulfils the search criteria, so it generates an alert, and 'Triggers Once' - an email is sent etc, the 'Job' completes

It is now 15 seconds past the minute - your RT search runs again, the result set STILL fulfils the criteria, so it triggers 'Once' again..

etc, etc

Better is to schedule your searches to run every minute looking back at the previous 60 seconds. (if you really must) but this kind of search does not work well with RT alerts, You 'could' use throttling to stop the same alert firing more than once, but you run the risk that two similar (but different) events could trigger resulting in throttling a message you care about.

The 'look back' method is better to make sure you don't get multiple alerts firing for a single event, and that each event gets alerted.

An even better approach is to widen the search window further, and look for events earliest=-2m, latest=-1m - This helps protect against messages which have been delayed by a few seconds and might otherwise be missed.

Better again is to run it over a longer time, like every 15 mins search -16min to -1m, but the immediacy is impacted

If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...