Alerting

How much of a delay is enough delay for an alert?

vanderaj2
Path Finder

I've read that a best practice for setting up a (non real-time) alert in Splunk is to schedule alerts with at least one minute of delay built in, to account for forwarding & indexing delays.

Well, I've got a alert setup to email me an alert whenever I've got a splunkd crashlog showing up anywhere in my environment. This alert runs every 5 minutes, with a 1 minute delay, like so:

Time range earliest: -6m@m latest: -1m@m
Cron schedule */5 * * * *
Condition if # of results > 0

However, I never get an email alert, even when Splunk finds results. Am I just not building enough delay into my alert? Is getting the right amount of delay just a matter of tweaking things until it works?

Thanks!

0 Karma
1 Solution

woodcock
Esteemed Legend

You almost certainly have bad timestamps in your data which are mis-labeling them so that events that really occurred "nowish" are being thrown hours into the future or the past. Install the Data Curator and Meta Woot apps and fix your _time problems. This is a deep topic and we do a TON of PS fixing this for clients. Those apps are by no means the whole story but they are a great first step. This is probably the single biggest (and most important) problem in the wild for Splunk (it is not a problem with the product; it is carelessness and confusion during the onboarding process).

View solution in original post

0 Karma

woodcock
Esteemed Legend

You almost certainly have bad timestamps in your data which are mis-labeling them so that events that really occurred "nowish" are being thrown hours into the future or the past. Install the Data Curator and Meta Woot apps and fix your _time problems. This is a deep topic and we do a TON of PS fixing this for clients. Those apps are by no means the whole story but they are a great first step. This is probably the single biggest (and most important) problem in the wild for Splunk (it is not a problem with the product; it is carelessness and confusion during the onboarding process).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...