Aggregate related logs into single log before index

[DEAD_BEEF]

My company's email system sends logs to my splunk instance, but does so in a less than ideal fashion. It sends logs for a message, broken up by each "phase" of the email. So a single email generates about 8-25 or so logs.

NOTE: The below log is after it has been aggregated together. Normally each line of the below log is intermixed with other email logs from multiple email servers.

Sample email generating 13 logs, aggregated via transaction

Start MID 1234 ICID 7890
MID 1234 SPF: helo identity postmaster@mail.us None
MID 1234 SPF: pra identity abc@bestnetwork.us None headers from 
MID 1234 Message-ID '<6e8905e4fb6c5340a7f4829ff5fc3d75@cust8-exchange.bestnetwork.us>'
MID 1234 ICID 7890 RID 0 To: <first.last@test.school.edu>
MID 1234 ICID 7890 From: <abc@bestnetwork.us>
MID 1234 SPF: mailfrom identity abc@bestnetwork.us None
MID 1234 Subject 'test log structure'
MID 1234 DMARC: Verification skipped (No record found for the sending domain)
Delivery start DCID 34567 MID 1234 to RID [0] to offbox Spam Quarantine
Message done DCID 34567 MID 1234 to RID [0] (external quarantine)
Message finished MID 1234 done
MID 1234 RID [0] Response 'ok: Message 12 accepted'

In a dashboard, I use a transaction on MID and email server (as there are many) to aggregate all of an email's logs into 1 log. What can I do to do this at index time? I'd like all the logs for a single message ID and server to be grouped together and indexed that way. Can Splunk do this or would the solution be external, say aggregating the logs with some script, saving to a .csv, then sending to the indexer?

[somesoni2 · Accepted solution]

If all your email logs, as seen in the sample event, start with phrase Start MID, you can setup correct LINE_BREAKER to merge all those lines together in a single event. May be something like this would work for you

props.conf on Heavy Forwarder or Indexer

[YourCustomSourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER =([\r\n]+)(?=Start MID)
...TIMESTAMP Extraction settings here..

[DEAD_BEEF]

What I have shown is what they come up as after I transaction them, they normally are jumbled up with other email logs from other email servers. As such, I don't think this will work. Do you have any other alternative suggestions?

[somesoni2]

AFAIK, creating an index time transaction is not supported. Also, the event breaking attribute I was talking about works on a source, so it'll be applied to all entries of a source file from single mail servers only (no cross-server overlapping of events), so may be worth a try.

[DEAD_BEEF]

I completely forgot that you specify by source... I was looking at the raw logs from all servers.

That being said, they don't all start with Start MID. That is one of the lines in the logs, but I noticed that they are often times in various orders. I think that's because the email server are not sending the logs with milliseconds, so many logs are all timestamped with the same seconds and hence jumbled, but it would make sense to start new messages with the Start MID line. I am betting if I can get ms added to the log entries, we will be good.

Otherwise, I think I am SOL for doing this in Splunk.