Alerting

Alert Trigger Question | Don't trigger it for that specific user for x amount of minutes

ycefalas
Loves-to-Learn Lots

I made an alert query that particularly looks for a windows failed login by users using stats. It works.

Whenever there is an event greater than 0, it’ll show case it and display it. It works.

Now here comes the problem:

The user who is constantly failing over a period of time also causes mass amount of alert notification triggers. Let’s say it’s every 10 minutes for the alert interval. Every 10 minutes we’ll be notified for the same user failing.

There is this option in splunk, that I am aware of:
alt text

This option works per say, however, if now a different user account were to have a +1 count, it will not be alerted because the alert won’t trigger until the next 20 minutes.

So here comes the question:

How can I make alert triggers intelligent enough to distinguish each user account as unique but if the user account was last seen then don’t trigger that same account for X amount of hours?

Hopefully I made sense, if not I’ll try to elaborate the problem further:

Account1 failed 5 logins at 1:00 triggered
Account2 failed 10 logins at 1:10 no trigger because of “after triggering the alert, don’t trigger it again for 20 minutes…”
Account1 failed 5 logins at 1:20 triggered

Tags (2)
0 Karma

DalJeanis
Legend

If you have a user who is "constantly failing" over a period of time, then that is a training problem.

If your job is running every 10 minutes, across a 10-minute timeframe, and deciding whether to alert, then you can just change it to run across a 20-minute timeframe, and alert only for those users that deserve an alert in the second 10 minutes but did not deserve (and therefore probably receive) an alert in the first 10 minutes. It won't be the splunk alert that's suppressing the long-term fails, but the search itself.across

You could actually go one further, just in case someone keeps failing long-term. Do the calculation across a 30 minute period. If the present period is an alert, suppress the alert only if the prior period was an alert but two periods ago was NOT an alert. Basically, if the guy has been failing for 30 minutes straight then there is something really wrong with him and we should send Mongo to go break his legs...

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...