Alerting

Why does my real-time alert continue to send emails for events that happened hours ago?

di2esysadmin
Path Finder

I have a simple search:

host=*prod*  "Too many open files"  source!="/opt/atlassian/jira-data/log/emh.log*"

I've set up a simple alert. Real-time, throttling on host for 1 minute. (pic attached)

We had 200+ of these errors, all within 2 seconds of each other, from a single host 2 hours ago. Since then, we've received no less than 24 emails alerting us of those events. I want one email, not 24.

What have I misconfigured?

Thanks!

alt text

0 Karma
1 Solution

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

View solution in original post

0 Karma

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

0 Karma

di2esysadmin
Path Finder

Thank you. I've made these changes. I've my fingers crossed this will do the trick.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...