Alerting

Why does my real-time alert continue to send emails for events that happened hours ago?

di2esysadmin
Path Finder

I have a simple search:

host=*prod*  "Too many open files"  source!="/opt/atlassian/jira-data/log/emh.log*"

I've set up a simple alert. Real-time, throttling on host for 1 minute. (pic attached)

We had 200+ of these errors, all within 2 seconds of each other, from a single host 2 hours ago. Since then, we've received no less than 24 emails alerting us of those events. I want one email, not 24.

What have I misconfigured?

Thanks!

alt text

0 Karma
1 Solution

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

View solution in original post

0 Karma

linu1988
Champion

Hi,
Please try to set the below setting with your preferred field. Always try to table to data you require. Moreover the throttling period is only 1 minute, so if you have events matching it will generate the alert. If you want hourly then throttle it to send 1 hour per host in the suppress triggering option.

alt text

Thanks,
L

0 Karma

di2esysadmin
Path Finder

Thank you. I've made these changes. I've my fingers crossed this will do the trick.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...