We have an alert that sends emails when triggered, and up until a few weeks ago it successfully sent emails to a company email and a PagerDuty alert email. We noticed that we stop getting the emails for this particular alert, and I want to investigate why. The alert still triggers a few times a day, but the emails never come. I've tested that the email addresses still work (for example, sending an email to the given PagerDuty email successfully creates an alert).
All of the docs and info I find about debugging this issue want me to traverse some Splunk log (for example, in $SPLUNK_HOME/var/log/splunk/scheduler.log). However, we use Splunk Cloud, so I don't have access to that log file. How can I find out why emails are not being sent?
... View more