Hi
I want to examine my mail server logfile. The logfile has many lines per logsession. Each line starts with a timestamp, then followed by a sessionid in [ and ]. Rest of line is data.
Each session is separated with ----------
Example lines
'Thu 2016-10-20 00:00:39: [39647:2] --> 250-ETRN
Thu 2016-10-20 00:00:39: [39647:2] --> 250-AUTH LOGIN CRAM-MD5 PLAIN
Thu 2016-10-20 00:00:39: [39647:2] --> 250-8BITMIME
Thu 2016-10-20 00:00:40: [39647:2] --> 250-STARTTLS
Thu 2016-10-20 00:00:41: [39647:2] --> 250 SIZE
Example Event separator:
Thu 2016-10-20 00:00:42: ----------
How can I make splunk Group all relevant lines into one event?
I'm running a Windows system with Splunk Light
This sounds like a good case to use the LINE_BREAKER setting in props.conf. Something like this
[your:sourcetype]
LINE_BREAKER = (\w{3}\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}:\s\-+)
MAX_TIMESTAMP_LOOKAHEAD = 30
This will tell splunk to use those separators as line breakers instead of the default \r\n. As a result, they won't show up in your data.
This sounds like a good case to use the LINE_BREAKER setting in props.conf. Something like this
[your:sourcetype]
LINE_BREAKER = (\w{3}\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}:\s\-+)
MAX_TIMESTAMP_LOOKAHEAD = 30
This will tell splunk to use those separators as line breakers instead of the default \r\n. As a result, they won't show up in your data.
Sorry to be a bit slow here.
I'm not sure where props.conf is located.
From the web interface of Splunk, I have added the wildcard to monitor, so this is located in apps/_server_app_mail/local/inputs.conf
There is not existing props.conf file. Should I just add one in this directory?
And so your regex deletes the timestamps?
I was kinda looking for something so I can group by sessionid ([39647:2]) - (whatever is in the brackets).
End goal is to list how many sessions has a specific line associated
e.g.
Fri 2016-10-21 12:40:01: [55444:3] * zen.spamhaus.org - failed - 127.0.0.4
Hope you can elaborate a bit. Thanks
To Recap: put the suggested lines on splunk Indexing server in $SPLUNK$/etc/system/default/props.conf
To modify the actual input I had to add a few lines on the input of the forwarding server as well. In my case in SplunkUniversalForwarder\etc\apps_server_app_mail\local\inputs.conf
I needed
crcSalt =
ignoreOlderThan=3d
maybe because i tried a few times and need to clear the data
No. Do not put this under /default/props.conf
You want to place this under /etc/system/local/props.conf
Hope I did this now. Thanks for your help
When you modify props.conf
, it will be on the indexer, not the forwarder where inputs.conf
is located. Your indexer is a central server where data is forwarded to, this is responsible for indexing and parsing the data, so it would make sense to put linebreaking rules where the data is parsed. Making changes to props.conf
will not work retroactively and will ONLY apply those changes to data being indexed after the changes were applied. So you have 3 options
1) Apply the changes to props.conf
and your older data will not be parsed correctly
2) Apply changes to props.conf
and verify your linebreaking is correct, then delete the old data and reindex your data with the changes applied
3) Use a transaction
or stats
command to combine the events at search time without having to set any line breaking rules on your indexer
Are you looking for a permanent solution to group these into a single event?
This is first time setting up the data source for splunk use. I'm looking for the best way to organize the data, so that queries will be as fast and easy to use in the future. I don't have any existing data in the Splunk server.
The Splunk data server and indexer is the same machine.
I'm looking for the location of the props.conf. Theres a lot in etc/apps and others in etc/system/default.
Which one is the correct file to change?
Ok this narrows it down, you should take suggestion 1 above, which is to set line_breaker
in your props.conf
The props.conf
file is located in %SPLUNK_HOME/etc/system/local
on the indexer.
Insert this stanza into your props.conf
file
[your:sourcetype]
LINE_BREAKER = (\w{3}\s\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2}:\d{2}:\s\-+)
MAX_TIMESTAMP_LOOKAHEAD = 30`
Replace "your:sourcetype" with the sourcetype name you specified in your inputs.conf
which is on the forwarder under %SPLUNK_HOME/etc/system/local/inputs.conf
After you've made these changes on your indexer, restart the splunk service and verify the linebreaking is correct
Ah just missed your post. Very good answer