We recently moved from a Single Indexer/Search head to Search head pointing to an indexer cluster and now I am finding that the "Alert Manager" Trigger action for alerts no longer functions properly. Triggered alerts do not appear to be making into the App defined "alerts" index. In know the "alerts" index works because I can shove events into it using the SPL "collect" command.
Please check the alert_manager.log and splunkd.log for any error messages.
What is the time window you're using for the alerts? After moving to multi-site clusters, our alerts that were running every 5 minutes were failing, and we had to push our detection windows back to look at events from -11m through -6m
instead of -5m through now()
.
The cluster is site local and the test alert I am using is a one time alert with a specific time in cron so I don't think it is a time window issue. This is really an issue with the "Alert Manager" App.
Ahh, gotcha.