Has anyone successfully parsed Exim logs into Splunk?

[ch1221]

I'm trying to get Exim logs parsed into Splunk to log inbound/outbound mail. I'm very new using RegEx and have been fighting to get something to parse it correctly. I've tried field-extractor and it only grabs about 13%, I've also tried the add-on builder.

Has anyone successfully consumed Exim logs? Can you provide some help?

[JeffLeshin]

I tried to answer this question a little earlier and it seems have failed somehow.
If this is ends up being a duplicate, my apologies.

I was looking for a head start on this myself when I found your post.
I have a new exim4 relay server that I need to monitor.

Here’s what I came up with myself - wiser spelunkers are welcome to improve on it:

First, a lot of the fields are parsed automatically by Splunk. That’s because they are name=value pairs. You can find their meanings in the exim4 docs. In Splunk they look like H=xxx, CN=xxx, etc.
You can use rename to give the fields friendlier names.

Here’s an example (your index name is undoubtedly different):

index=smtpexim | rename H as HostSender | stats count by HostSender host

This is useful for seeing which hosts are using the relay server.

I also needed to get alerts when there are email transfer failures. Here I needed to combine all the events related to one mail transfer. A good use of transactions. I also had to extract the message id which, strangely enough, is not part of a name value pair. It follows the time stamp in the log.

This search uses the index and sourcetype I defined. Just substitute here.
In actual use I saved the regex as a field extraction after testing it in the search, below.

index=smtpexim  sourcetype=exim4logs | rex field=_raw "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\s(?P<messageID>.+?)\s.*" |transaction messageID | search NOT Completed AND NOT "queue run"

This should not return any events unless a message transfer is not successfully completed. I still haven’t tested it with a bogus message. But I think the Splunk part is good start.

[JeffLeshin]

I was looking for a head start on this myself when I found your post.
I have a new exim4 relay server that I need to monitor.

Here’s what I came up with myself - wiser spelunkers are welcome to improve on it:

First, a lot of the fields are parsed automatically by Splunk. That’s because they are name=value pairs. You can find their meanings in the exim4 docs. In Splunk they look like H=xxx, CN=xxx, etc.
You can use rename to give the fields friendlier names.

Here’s an example (your index name is undoubtedly different):

index=smtpexim | rename H as HostSender | stats count by HostSender host

This is useful for seeing which hosts are using the relay server.

I also needed to get alerts when there are email transfer failures. Here I needed to combine all the events related to one mail transfer. A good use of transactions. I also had to extract the message id which, strangely enough, is not part of a name value pair. It follows the time stamp in the log.

This search uses the index and sourcetype I defined. Just substitute here.
In actual use I saved the regex as a field extraction after testing it in the search, below.

index=smtpexim  sourcetype=exim4logs | rex field=_raw "^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\s(?P<messageID>.+?)\s.*" |transaction messageID | search NOT Completed AND NOT "queue run"

This should not return any events unless a message transfer is not successfully completed. I still haven’t tested it with a bogus message. But I think the Splunk part is good starting point.

[JDukeSplunk]

Could you provide some sample data?

[ch1221]

Here is something similar to what I am working with. As you can see, it includes multiple events in the logs and once those can be separated out, combining the messages with transaction should be straightforward.

2017-06-22 00:00:32 1dNw6R-0004dN-VU <= Yajane_doe@generic.com H=svr-xxx--01.xxx.genericg.com [99.99.99.999] P=esmtp K S=76706 id=d7a4493e0b7e40d29ee9156d95ee9f02@svr-xxx--01.xxx.genericg.com
2017-06-22 00:00:32 H=esa3.generic.iphmx.com [99.99.999.999] F= rejected RCPT : bounce_localunknown router forced verify failure
2017-06-22 00:00:32 1dNw6Q-0004bW-Im => mumad.atif.b.abd.raz@xx.com.com R=dnslookup T=remote_smtp H=mga14.xx.com.com [99.99.99.999] X=TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 ok:  Message 227984017 accepted"
2017-06-22 00:00:32 1dNw6Q-0004bW-Im -> amit.radhak@xx.com.com R=dnslookup T=remote_smtp H=mga14.xx.com.com [99.99.99.99] X=TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 ok:  Message 227984017 accepted"
2017-06-22 00:00:32 1dNw6Q-0004bW-Im -> abhil.bm@xx.com.com R=dnslookup T=remote_smtp H=mga14.xx.com.com [99.99.99.999] X=TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 ok:  Message 227984017 accepted"
2017-06-22 00:00:32 1dNw6Q-0004bW-Im -> hish.mar.m@xx.com.com R=dnslookup T=remote_smtp H=mga14.xx.com.com [99.99.99.999] X=TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256 CV=yes C="250 ok:  Message 227984017 accepted"
2017-06-22 00:00:32 1dNw6Q-0004bW-Im Completed
2017-06-22 00:00:32 1dNw6R-0004dN-VU => arn.sine@nmt.com R=dnslookup T=remote_smtp H=aspmx.l.gxxx.com [99.99.99.999] X=TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128 CV=yes K C="250 2.0.0 OK g6si645764iof.8 - gsmtp"
2017-06-22 00:00:32 1dNw6R-0004dN-VU Completed
2017-06-22 00:00:33 no IP address found for host aj.to.genericg.com (during SMTP connection from es3.generic.iphmx.com [99.99.999.999])
2017-06-22 00:00:33 H=esa3.generic.iphmx.com [99.99.999.999] X=TLSv1.2:RC4-SHA:128 CV=no F= temporarily rejected RCPT : lookup of host "aj.to.generic.com" failed in xxx_routes router
2017-06-22 00:00:34 1dNw6U-0004dl-Ia <= error@err.gna.co.jp H=esa2.generic.iphmx.com [99.99.99.99] P=esmtps X=TLSv1.2:RC4-SHA:128 CV=no S=12467 id=1498.114822.305392@err.gna.co.jp
2017-06-22 00:00:35 1dNw6U-0004dl-Ia => masashi_shigemori@generic.com R=exchange_users T=remote_smtp_ex_hosts H=mail-na.genericg.com [99.99.999.999] X=TLSv1:ECDHE-RSA-AES256-SHA:256 CV=yes K C="250 2.6.0 <1498.114822.305392@err.gna.co.jp> [InternalId=5521513] Queued mail for delivery"
2017-06-22 00:00:35 1dNw6U-0004dl-Ia Completed

[kmorris_splunk]

Ingesting this sample through the GUI, seems to parse the events OK. It's also grabbing the timestamp properly. A good practice would be to take a sample, ingest it through the GUI. If things aren't linebreaking or timestamping properly, you can make adjustments in the GUI and save the settings in a sourcetype. Once you have the sourcetype, you can create your input to read the logs in, specifying the new sourcetype you created.