Alerting

Why am I not always receiving emails for a scheduled alert that I can see is triggered locally in Splunk?

gyslainlatsa
Motivator

hello all,

I scheduled an alert to send me emails after every 1h as soon as the word "error" is found in the events, but I do not always manage to receive emails, yet alert is triggered in Splunk locally. Can someone help me to configure so that I can receive mails?
any help will be welcome.
thank you in advance !!!!

Tags (2)
0 Karma

jnicholsenernoc
Path Finder

Here are a few ways to troubleshoot alerts not being delivered consistently via email.

Please find savedsearches.conf on the file system that contains this search and post the entire stanza if possible. You may be throttling and so forth, so lets look at the configuration as well.

First, check your SPAM folder and any upstream email SPAM firewall.

Then:
1. Verify the results of the scheduled searches by clicking Settings->Searches, Reports, Alerts
2. Find your search in the list, click "View Recent"
3. You may need to adjust the drop downs to filter to your search
4. Inspect if the "Dispatched At" column is per the frequency you have configured in the alert (the scheduler may be having issues firing it on the frequency you select, for example if search runs every 5 minutes and takes 6 minutes to run, errors on the peer, etc)
5. Click on the name of the Search to view the execution's results, does it look like you expect to trigger the alert?
6. If you are on splunk 6.x, you can view the Triggered Alerts list directly under Activity->Triggered alerts

Finally:
If all of the above looks fine and you are triggering the emails correctly, you then need to investigate the email server settings under Settings->Server Settings->Email Settings.

If you are running linux, log into the host and inspect /var/log/maillog, this will show if the host is queuing the mail for delivery, if it is being rejected downstream in a SPAM filter and so forth. If you see issues here, let us know, but google it and troubleshoot it as a mail issue as it is down stream of splunk.

0 Karma

echalex
Builder

It would help if you posted the configuration of your alert. Are you perhaps using throttling?

0 Karma

gyslainlatsa
Motivator

please give me only the configuration of the Emails settings in splunk for:
Mail host =
Email security =
Username =
Password =

0 Karma

echalex
Builder

Do you mean to say that you have not configured these settings? These are system-level settings, so for example the mail host needs to correspond to your mailer's host address. The rest of these settings may or may not be needed, depending on your mailer's settings. Contact your email administrator.

But I really meant the settings of the alert itself. Like: what is the scheduling? What are the actions? Do you have a condition for the alert?

0 Karma

gyslainlatsa
Motivator

the alert will be triggered after 1h whose time range is defined between -1h and now.
theschedule type isbasic and Runto the value Every hour. and the condition of the alertif number of events is Greater Than 0 and alertmode once per searchand Alert actions I checked send email to enable. To and I put the recipient's email. but we can not receive alerts.

0 Karma
Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...