How to troubleshoot why I am not receiving emails for some scheduled alerts and reports after upgrading to Splunk 6.2?

[hartfoml]

I have my search head set up to send email alerts and reports. I am getting some alerts, but after upgrading to Splunk 6.2, some of my alerts are not working and I cannot seem to get new reports working.

Where can I go to see what the problem is with the emailing of the reports and alerts?

I can see the report after it runs, but it just never shows up in the email.

I am getting some alerts and reports in email.

I even tried to set up new reports and alerts after running successful searches, but still no report for some new reports while others are successful.

I just need to know were to look to find out if it is a syntax issue or if it is being blocked somehow at the email server.

[hartfoml · Accepted solution]

So here is the answer that I got while working with support;

Use this search:

Your_search Foo=* | stats count by "foo" | sort "foo"| sendemail to=yourname@yourdomain.com server=mail_relay.yourdomain.com subject="Here is an email from Splunk SearchHead" message="This is an example message" sendresults=true inline=true format=raw sendpdf=true

Change sendprf to sendcsv or change true to false to see what is working and what is not.

In my case everything but CSV attachment was working. this lead support to look at the sendmail.py and why all other functions were working and not send mail.

I had added a datestamp variable to be added to the csv file name like this:

# create datestamp for filename
datestamp = time.strftime('%Y-%m-%d')

I don't remember where I got this Mod but it came with this line also

# strip control characters, forward & backslash
filename = re.sub(r'[\x00-\x1f\x7f/\\]+', '-', filename)

The I changed this line in the sendmail.py file in $splunkhome/etc/apps/search/bin/
from: csvAttachment.add_header('Content-Disposition', 'attachment', filename="splunk-results.csv")
to: csvAttachment.add_header('Content-Disposition', 'attachment', filename = '%s-%s.csv' % (subject, datestamp))

Support was able to see that the filename= was referenced in the "# strip control caterers" before it was referenced in the "csvAttachment.add_header"

I commented out the line filename = re.sub(r'[\x00-\x1f\x7f/\\]+', '-', filename) and attachments started working again.

At some time in the future I will test taking out control characters but for now it is working.

use the email test above to test email functions when editing custom sendmail.py configurations

[paulstark]

to help troubleshoot ... i would create the report then "| sendemail to=blah@hotmail.com"
you will see any errors in mail config there. It won't tell you if its being blocked, but if it completes without error. recently, i have seen the requirement for a host fqdn in $splunk_home/etc/system/local/alert_actions.conf

[hartfoml]

Thanks @paulstark and @davebrooking for helping,

I got this search from support:
index="firewall" source_zone_name="*" destination_zone_name="*"
policy_name="*" | stats count by "source_zone_name",
"destination_zone_name", "policy_name" | sort "source_zone_name",
"destination_zone_name", "policy_name"| sendemail
to=hartfoml@mydomain.com server=mrelay.mydomain.com subject="Here is an
email from Splunk SH1" message="This is an example message"
sendresults=true inline=true format=raw sendpdf=true

This search did send an email with an attachment BUT when I changed the end to sendcsv=true /i did not get the email

I sent a new Diag to support and have not heard back. I am getting python errors that read ERROR sendemail:1213 - local variable 'filename' referenced before assignment at the time that the email is suppose to have been sent.

Thanks again for helping

[hartfoml]

Thanks for the help Paul,

Sorry for the slow responce:

This is what I did:

index=MyIndex Source=MySource | Stats Count By Source, Destination, Policy | sort Source, Destination, Policy | sendmail to=My.Address@MyDomain.com

This did not run, and errored out with "Unknown search command 'sendmail'.
ssooo, I am guessing that this is not what you meant for me to try.

[davebrooking]

If you read Paul's search the final command in the search is sendemail and not sendmail as you have used.

[paulstark]

build your query then "| sendemail to=me@domain.com" ....

[hartfoml]

I came across this search from Martin_Mueller index=_internal sourcetype=scheduler alert_actions=*email*

This doesn't tell me much or if the report was actually emailed