Splunk Search

How to extract mail logs (clearswift) and link across multiple lines

lmaclean
Path Finder

Hi,

I have searched and haven't really found anything to parse Clearswift mail logs. The issue is that one email may be on 10+ log lines because each part of the email header is on it's own line. On top of that I need to filter the logs based upon the log type which is the 9th field within the log if the mail system or 6th if another part of the clearswift system. Though for the email logs there is the transaction ID (example below "t6T5CBOe007210") which can be used to link the lines together but I am just not sure on how to do this.

An example of the logs:

Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: TLS_Requested=0 (none)
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: Not using TLS to deliver to smtp server: gmail-smtp-in.l.google.com
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> MAIL From:<email> SIZE=274311
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: <-- 250 2.1.0 OK fl3si6695494pad.107 - gsmtp
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> RCPT To:<email>
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptTrustManagerBad: true
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptAntiSpoofing: true

Jul 29 15:27:53 <host> 1 2015-07-29T05:26:49Z <mailserver> pmm - - - INFO : RemoteActionWatcherTask: 1 action files copied to <mailserver>

skoelpin
SplunkTrust
SplunkTrust

Can you explain what you mean by filtering. Do you mean linebreaking an event?

0 Karma

lmaclean
Path Finder

I mean creating different tags or sourcetypes based upon if it is part of the "mail" system then for "sm-outbound" events group them together by the transaction ID, if "sm-inbound" same thing, etc... While if it was for another system like "pmm" then having a sourcetype to read those lines differently.

0 Karma

domenico_perre
Path Finder

Hi lmaclean,

Creating sourcetypes can be simple but it depends on how you are logging and where you are logging to.

Are you logging to a file or are you sending straight syslog to your splunk indexer. Do you have any intermediate forwarders?

I can give you a hand with creating an app if you want that will set correct sourcetypes etc. Its not too hard once you do a couple :).

A props.conf could create a transaction id field on the fly for you. Can you post what your sourcetype is looking like at the moment?

props.conf (on your search head) could be something like this if your sourcetype was [clearswift]
[clearswift]
EXTRACT-transaction-id = (?:\s+)(?[\d\w]{14})

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...