How to extract mail logs (clearswift) and link across multiple lines

[lmaclean]

Hi,

I have searched and haven't really found anything to parse Clearswift mail logs. The issue is that one email may be on 10+ log lines because each part of the email header is on it's own line. On top of that I need to filter the logs based upon the log type which is the 9th field within the log if the mail system or 6th if another part of the clearswift system. Though for the email logs there is the transaction ID (example below "t6T5CBOe007210") which can be used to link the lines together but I am just not sure on how to do this.

An example of the logs:

Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: TLS_Requested=0 (none)
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: Not using TLS to deliver to smtp server: gmail-smtp-in.l.google.com
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> MAIL From:<email> SIZE=274311
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: <-- 250 2.1.0 OK fl3si6695494pad.107 - gsmtp
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> RCPT To:<email>
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptTrustManagerBad: true
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptAntiSpoofing: true

Jul 29 15:27:53 <host> 1 2015-07-29T05:26:49Z <mailserver> pmm - - - INFO : RemoteActionWatcherTask: 1 action files copied to <mailserver>

[skoelpin]

Can you explain what you mean by filtering. Do you mean linebreaking an event?

[lmaclean]

I mean creating different tags or sourcetypes based upon if it is part of the "mail" system then for "sm-outbound" events group them together by the transaction ID, if "sm-inbound" same thing, etc... While if it was for another system like "pmm" then having a sourcetype to read those lines differently.

[domenico_perre]

Hi lmaclean,

Creating sourcetypes can be simple but it depends on how you are logging and where you are logging to.

Are you logging to a file or are you sending straight syslog to your splunk indexer. Do you have any intermediate forwarders?

I can give you a hand with creating an app if you want that will set correct sourcetypes etc. Its not too hard once you do a couple :).

A props.conf could create a transaction id field on the fly for you. Can you post what your sourcetype is looking like at the moment?

props.conf (on your search head) could be something like this if your sourcetype was [clearswift]
[clearswift]
EXTRACT-transaction-id = (?:\s+)(?[\d\w]{14})