Splunk consuming email alerts from other monitoring tools

[jhally]

Hi All,

does anyone know of an app or configuration to consume email alerts generated by other monitoring tools. it seems that every monitoring tool has the ability to send alerts via email, but none seem to be able to receive the email and consume the information, at least not easily.

the thought is to simply add a splunk email address that gets sent to the splunk server/collector and then be able to splunk on the data provided.

thnoughts?

thanks!

[mattymo]

Inspired by duckfez's advice, here is how I caught CSVs sent to Splunk> on Ubuntu 16.04LTS using fetchmail procmail and uudeview

Install fetchmail procmail and uudeview

mattymo@n00bserver:~$apt install fetchmail procmail uudeview

Create directories in your $HOME

mattymo@n00bserver:~$mkdir mail_backup
mattymo@n00bserver:~$mkdir mail_attachments
mattymo@n00bserver:~$mkdir mail_logs

Create .fetchmailrc in $HOME and configure

mattymo@n00bserver:~$vi .fetchmailrc

Copy & Paste:

####OCTOBER 17 2016 - mattymo in the n00blab making Splunk> eat csv sent by other systems
#http://www.fetchmail.info/fetchmail-man.html
#Big up Falko Timme - https://www.howtoforge.com/debian_etch_fetchmail
#Big up Thomas Kuther - https://kuther.net/howtos/howto-receive-mail-and-save-attachment-fetchmail-procmail-and-metamail

set syslog 
set logfile "mail_logs/fetchmail.log"
set postmaster "mattymo"
set daemon 300

poll <yourmailserver> proto POP3 port <yourport>   
    user "<youremailaddress>" there with password "<yourpassword>" is <youruser> here 

mda '/usr/bin/procmail -d %T'

Lock fetchmailrc down

mattymo@n00bserver:~$chmod 600 .fetchmailrc

Configure .procmailrc

mattymo@n00bserver:~$ vi .procmailrc

Copy & Paste

#OCTOBER 17 2016 - mattymo in the n00blab making Splunk> eat csv sent by other systems
#https://linux.die.net/man/5/procmailrc
#Big up duckfez https://answers.splunk.com/answers/50048/splunk-consuming-email-alerts-from-other-monitoring-tools.h...
#Big up Thomas Kuther - https://kuther.net/howtos/howto-receive-mail-and-save-attachment-fetchmail-procmail-and-metamail

#Log to mail_logs
LOGFILE=$HOME/mail_logs/.procmail.log
VERBOSE=yes

:0
*^content-Type:
{

   # backup the complete mail first..
   # you can leave out this part if you don't want a backup of the complete mail
   :0c:
   $HOME/mail_backup

   # Now the actual unpacking part
   # forward to uudeview and unpack attachments to $HOME/attachments
   :0fw
   | uudeview -p $HOME/mail_attachments -
}

Now on to crafting an inputs.conf to only consume .csv$ from the mail_attachments and playing with more email settings and scenarios! Will update here as I go!

Feedback welcome!

[dwaddle]

I've not done it, but this should be near-trivial on any linux server that uses procmail as a delivery agent. On the Splunk side, configure a batch input in inputs.conf as follows:

[batch:///home/splunk/mailqueue]
move_policy = sinkhole
sourcetype = inbound_mail

In props.conf, set up a LINE_BREAKER to make the whole-file a single event:

[inbound_mail]
LINE_BREAKER=([\r\n]+)3flkjfjkfielkfjlkdfiewfkfdkfkfikds3288nbcnmc032jckcewhjk43847rf

Basically, setting a "statistically unlikely to occur" LINE_BREAKER...

Then, in the splunk user's .procmailrc:

LOGFILE=$HOME/.procmail.log
VERBOSE=yes

:0
* Subject: .*
$HOME/mailqueue

With this result, procmail should take each inbound message and put it in a unique file in $HOME/mailqueue. These will be picked up by Splunk and indexed as whole files. You can tune the .procmailrc to only index certain subjects, or messages from certain places - procmail is very capable.

[DrewO]

There is a Splunk App for importing email via IMAP here:

http://splunk-base.splunk.com/apps/22390/splunk-for-imap

[jhally]

Thanks!

I was looking for something that wasn't pulling from a mail server but consuming the mail that gets sent to a server. For example, spinning up sendmail on the splunk server that receives mail for splunk@somehost.com and then locally consuming the data. I'll check into this though.

thanks again!