How can I merge multiple mail log lines into one event

[lars_meldgaard]

Hi

I want to examine my mail server logfile. The logfile has many lines per logsession. Each line starts with a timestamp, then followed by a sessionid in [ and ]. Rest of line is data.
Each session is separated with ----------

Example lines
'Thu 2016-10-20 00:00:39: [39647:2] --> 250-ETRN
Thu 2016-10-20 00:00:39: [39647:2] --> 250-AUTH LOGIN CRAM-MD5 PLAIN
Thu 2016-10-20 00:00:39: [39647:2] --> 250-8BITMIME
Thu 2016-10-20 00:00:40: [39647:2] --> 250-STARTTLS
Thu 2016-10-20 00:00:41: [39647:2] --> 250 SIZE

Example Event separator:
Thu 2016-10-20 00:00:42: ----------

How can I make splunk Group all relevant lines into one event?
I'm running a Windows system with Splunk Light

[maciep · Accepted solution]

This sounds like a good case to use the LINE_BREAKER setting in props.conf. Something like this

[your:sourcetype]
LINE_BREAKER = (\w{3}\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}:\s\-+)
MAX_TIMESTAMP_LOOKAHEAD = 30

This will tell splunk to use those separators as line breakers instead of the default \r\n. As a result, they won't show up in your data.

[lars_meldgaard]

Sorry to be a bit slow here.

I'm not sure where props.conf is located.
From the web interface of Splunk, I have added the wildcard to monitor, so this is located in apps/_server_app_mail/local/inputs.conf

There is not existing props.conf file. Should I just add one in this directory?

And so your regex deletes the timestamps?

I was kinda looking for something so I can group by sessionid ([39647:2]) - (whatever is in the brackets).
End goal is to list how many sessions has a specific line associated
e.g.
Fri 2016-10-21 12:40:01: [55444:3] * zen.spamhaus.org - failed - 127.0.0.4

Hope you can elaborate a bit. Thanks

[lars_meldgaard]

To Recap: put the suggested lines on splunk Indexing server in $SPLUNK$/etc/system/default/props.conf
To modify the actual input I had to add a few lines on the input of the forwarding server as well. In my case in SplunkUniversalForwarder\etc\apps_server_app_mail\local\inputs.conf

I needed
crcSalt =
ignoreOlderThan=3d
maybe because i tried a few times and need to clear the data

[skoelpin]

No. Do not put this under /default/props.conf

You want to place this under /etc/system/local/props.conf

[lars_meldgaard]

Hope I did this now. Thanks for your help

[skoelpin]

When you modify props.conf, it will be on the indexer, not the forwarder where inputs.conf is located. Your indexer is a central server where data is forwarded to, this is responsible for indexing and parsing the data, so it would make sense to put linebreaking rules where the data is parsed. Making changes to props.conf will not work retroactively and will ONLY apply those changes to data being indexed after the changes were applied. So you have 3 options

1) Apply the changes to props.conf and your older data will not be parsed correctly
2) Apply changes to props.conf and verify your linebreaking is correct, then delete the old data and reindex your data with the changes applied
3) Use a transaction or stats command to combine the events at search time without having to set any line breaking rules on your indexer

Are you looking for a permanent solution to group these into a single event?

[lars_meldgaard]

This is first time setting up the data source for splunk use. I'm looking for the best way to organize the data, so that queries will be as fast and easy to use in the future. I don't have any existing data in the Splunk server.
The Splunk data server and indexer is the same machine.
I'm looking for the location of the props.conf. Theres a lot in etc/apps and others in etc/system/default.
Which one is the correct file to change?

[skoelpin]

Ok this narrows it down, you should take suggestion 1 above, which is to set line_breaker in your props.conf

The props.conf file is located in %SPLUNK_HOME/etc/system/localon the indexer.

Insert this stanza into your props.conf file

[your:sourcetype]
 LINE_BREAKER = (\w{3}\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}:\s\-+)
 MAX_TIMESTAMP_LOOKAHEAD = 30`

Replace "your:sourcetype" with the sourcetype name you specified in your inputs.conf which is on the forwarder under %SPLUNK_HOME/etc/system/local/inputs.conf

After you've made these changes on your indexer, restart the splunk service and verify the linebreaking is correct

[lars_meldgaard]

Ah just missed your post. Very good answer