Why do I see no triggered alerts for an alert that should definitely be triggering an alert?

[leejeason]

I have a simple search:

sourcetype=iis sc_status=500

The search returns results. I saved the search as an alert. The alert is cron scheduled to run every minute (Earliest: -1m@m, Cron Expression: */1 * * * *). The only condition on the alert is that results must be greater than 0.

When I open the alert in search, it gives results. When I look at the jobs page, I clearly see it running the alert search. Further, the jobs page clearly shows that many of these entries have positive result counts. When I inspect the job, I see the alert settings all look valid and resultCount is indeed a positive number.

However, the triggered alerts page shows nothing - not a single entry there. So what am I missing? Any tips on how to troubleshoot something like this?

[frobinson_splun · Accepted solution]

Hi @leejeason,
Did you happen to set up the "Add to Triggered Alerts" alert action, for this particular alert? Or other alert actions? If not, this might explain some of the behavior you're seeing. If the "add to triggered alerts" action is not enabled, then the alert triggering instances won't be listed on the "Triggered Alerts" page.

Here is some documentation that might help:
http://docs.splunk.com/Documentation/Splunk/6.3.1/Alert/Triggeredalertaction
http://docs.splunk.com/Documentation/Splunk/6.3.1/Alert/Reviewtriggeredalerts

Let me know if you have other questions or are still seeing this behavior after double-checking alert action configuration.
@frobinson_splunk

[leejeason]

Yep, that was it! I simply missed that there was an explicit action to do that and just assumed they'd end up there automatically. Thanks so much for the clarification!

[frobinson_splun]

Oh, glad this helped!
Cheers,
@frobinson_splunk