Getting Data In

Some Single Line Messages Are Merged into a Single Event

fandingo
New Member

I'm working with data that looks like this:

QA4 :: 1354371771 :: 020_grid_progress :: M020_grid_progress :: alert :: Grid recovery completed on Sat Dec 1 09:22:49 2012: There were 17 active application(s) when the grid controller went down. 3 application(s) have been recovered. The state of 11 applications has been reacquired.3 application(s) failed to be recovered. See the controller system log for details. QA4 :: 350399612 :: 050_filer_status :: M050_filer_status :: info :: Internal condition 'filer status' occurred. This condition should not affect the operation of your grid. Please notify support that this error has occurred and reference SCR2301.

Each event ends with a UNIX newline (\n), and I've verified that the newline is always properly set.

The weird part is that Splunk sometimes merges events. Here is how Splunk has interpreted the data. I used the JSON export from Splunk because it shows the newline character.

{"preview":false,"result":{"raw":"QA4 :: 1354382431 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:20:30 2012. Volume maintenance is required. Found 8 unused volumes.\nQA4 :: 1354370459 :: 500_3tctlmon_report :: M500_3tctlmon_report :: alert :: Controller restarted on Sat Dec 1 09:00:10 2012 because of an unexpected shutdown. Please note that this failure has no effect on the applications that may be running on the grid. Please contact technical support. ","_time":"2012-12-01T12:20:30.000-0600","date_hour":"12","date_mday":"1","date_minute":"20","date_month":"december","date_second":"30","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"2","punct":"::::::::::______::..._:::","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}

{"preview":false,"result":{"raw":"QA2 :: 1354382375 :: 070_vol_maint :: M070_vol_maint :: info :: Volume check completed on Sat Dec 1 12:19:34 2012. Volume maintenance is required. Found 74 unused volumes.","_time":"2012-12-01T12:19:34.000-0600","date_hour":"12","date_mday":"1","date_minute":"19","date_month":"december","date_second":"34","date_wday":"saturday","date_year":"2012","date_zone":"local","host":"dc2opsdashqa01","index":"main","linecount":"1","punct":"::::::::::______::..__.","source":"/var/log/applogic-dashboard-messages.log","sourcetype":"applogic-dashboard-msg","splunk_server":"dc2mgmtsplqa01","timeendpos":"114","timestartpos":"94"}}

Notice how the first event actually includes two. (Look for "\nQA4" in it.)

Why has Splunk combined the first two messages, but properly splits the third one into a separate event? Is there anything I can do to force a split on "\n"?

Thanks,

Tags (1)
0 Karma

yannK
Splunk Employee
Splunk Employee

setup a sourcetype for your events, that disable the multiline detection.
in prop.conf

[mysourcetype]
SHOULD_LINEMERGE=false

see http://docs.splunk.com/Documentation/Splunk/5.0.1/Data/IndexMulti-lineEvents

fandingo
New Member

I have been clearing the data every time, and the re-indexed messages aren't affected. I've also run the data through "| sort -R" on the shell before Splunk picks it up. Each time, it's completely different messages that are merged, so there's nothing weird happening with the line endings.

0 Karma

lguinn2
Legend

Once Splunk has indexed data, it will not change it. So you will need to clean the events from the index and re-index the source data in order to make the changes.

./splunk clean eventdata -index yourindex

will do the trick - although Splunk will re-index everything in that index and this might be an issue for your license.

fandingo
New Member

We only have a single indexer, and these logs are only present on one server.

I worked some with engineers in efnet and this updated props does not work either. (I modified the log format to have the epoch timestamp first.)

etc/users/admin/search/local/props.conf

[applogic-dashboard-msg]
SHOULD_LINEMERGE=false
TIME_FORMAT=%s
EXTRACT-timestamp-grid-id-name-severity-text = ^[0-9]+ :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^\n]+)

0 Karma

yannK
Splunk Employee
Splunk Employee

Do you have multiple forwarders and indexers ?
The props.conf has to be on the indexer (for index time parameters)

0 Karma

fandingo
New Member

Thanks for the reply, but that did not fix the problem. My props.conf is now:

[applogic-msg]
SHOULD_LINEMERGE=false
EXTRACT-grid-timestamp-id-name-severity = ^(?P[^ ]+) :: (?P[0-9]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^ ]+) :: (?P[^\n]+)

I appended the messages from earlier to this file, but some of them (including the example in my question) are still merged.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...