Getting Data In

Has anyone used Splunk to indirectly monitor mail flow (ie verify a Cloud provider isn't down)?

robtheorg
New Member

Recently deployed a Cloud-based email gateway, and after a couple of hiccups, now being asked to put some monitoring in place to verify that msgs are stacked up/delayed going through the Cloud provider.

Its not a challenge to create a script to fire off email's on an interval, but then validating that the msgs got to the recipient mailbox in a timely manner could be tricky.

Splunk is already in the environment (handling the SMTP gateway logs, the middle of the mail flow).

Has anyone already done something like this w/ Splunk, or would I be better served w/ a dedicated monitoring tool?

Thanks,

Robert

Tags (3)
0 Karma

ahall_splunk
Splunk Employee
Splunk Employee

There are a couple of methods to do this, but they generally require a custom scripted input. My favorite is to create an email and send via direct SMTP to the cloud provider, directed at a mailbox inside your organization that drops mail on the floor. Your email should include a well-known subject (e.g. [TIMING TEST] ). Make sure you include the timezone in the date/time string for proper evaluation later on.

In your SMTP logs, you do a search where the recipient is your dropbox and the subject includes [TIMING TEST]. If you are using the Splunk App for Microsoft Exchange, that search is:

eventtype=msexchange-msgtrack source_id=SMTP event_id=RECEIVE recipient="mydropbox@domain.com" subject="*[TIMING TEST]*"

Now, you can do two things here. Firstly, you can calculate the delay - simply extract the date/time that you sent the message from the subject, convert it to a timestamp with eval and strptime(), then subtract the timestamp from _time to get the # seconds between when you sent and when you received it.

Secondly, you can detect when mail is stacking up by checking for the last time you received a timing test. If you send your timing test every 5 minutes and your delivery SLA is 1 minute, you can do an alert if you have not received a message within 6 minutes. I'd add a few seconds to that for delays in logging, so your search looks like:

eventtype=msexchange-msgtrack source_id=SMTP event_id=RECEIVE recipient="mydropbox@domain.com" subject="*[TIMING TEST]*"|eval td=time()-_time|where td > 375|table subject,td

Search over the last 10 minutes and convert to an alert.

I've obviously taken examples from the Splunk App for Microsoft Exchange here, but you could adjust for any SMTP gateway.

bmacias84
Champion

With your clould provider what method would you use verify stacked up or delayed messages?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...