Alerting

Why is my scheduled alert not getting triggered?

dragon0087
New Member

My setup is simple. Splunk reads a file /var/log/snmp.log into index "snmp". I created a search: index="snmp" and created an alert for this. It is scheduled to search for results in the last minute every 1 minute and should trigger if the count>1. In the action, I send the alert to alert manager, but it's not working. I could see the search completed on jobs, but no alert showed in triggered alerts nor alert manager.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You might have over 1 minute of indexing / event latency.

Try this in your search instead:

index=snmp _index_earliest=-1m

This will go by the time it was indexed and not by the date timestamp in the events.

It is a best practice to have a "trailing" search for these types of alerts. Something like this would search for the minute of time from 10 minutes ago, and would be more likely to find the events that are latent.

index=snmp earliest=-11m@m latest=-10m@m

Play with _index_earliest, _index_latest, earliest and latest a little and I think you'll solve your problem.

Here's a good write up on the issue I think you're facing:
https://answers.splunk.com/answers/11870/how-can-i-view-the-indexing-latency-for-incoming-events-in-...

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...