Why is Splunk not breaking each log line into single events?

[aa123s]

Hello,

After being loaded into Splunk, my event looks like this:

EVENT BEGINNING
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered layouts/partials/_analytics.html.erb (10.5ms)
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered meta/components/_header_mobile.html.erb (0.4ms)
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered meta/components/_menu_mobile.html.erb (4.5ms)
[dd61c495-caf3-4d07-bfb5-8421c8ade35b]   Rendered layouts/partials/_google_tag_manager.html.erb (0.1ms)
EVENT END

My event ID's are enclosed between square brackets. As you can see within the same event I have multiple logfile lines. That's a problem because when I try to filter by using 'transaction', multiple transactions id's are mixed together (in the above example 3c58db35-1eef-43a5-8b57-57081bec264c and dd61c495-caf3-4d07-bfb5-8421c8ade35b).
Instead, I would like to have a single line per event so that It would be possible to correctly filter out transactions ids. Like so:

EVENT BEGINNING
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered layouts/partials/_analytics.html.erb (10.5ms)
EVENT END

EVENT BEGINNING
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered meta/components/_header_mobile.html.erb (0.4ms)
EVENT END

EVENT BEGINNING
[3c58db35-1eef-43a5-8b57-57081bec264c]   Rendered meta/components/_menu_mobile.html.erb (4.5ms)
EVENT END

EVENT BEGINNING
[dd61c495-caf3-4d07-bfb5-8421c8ade35b]   Rendered layouts/partials/_google_tag_manager.html.erb (0.1ms)
EVENT END

Shouldn't Splunk, by Default, split lines right after each \r\n ? (that's the LINE_BREAKER default value). Somehow instead, for reasons that I don't understand, Splunk is grouping lines that are unrelated to one another.

Thank you for your help

Alain

[somesoni2 · Accepted solution]

I'm guessing you don't have any event parsing configuraton for your sourcetype. The default LINE_BREAKER is [\r\n]+ but that only defines the line breaking. There are other attributes which define the line merging and default values of other attributes are causing this merge of line into single events. See the below link for more details on how Splunk breaks the events.

https://docs.splunk.com/Documentation/Splunk/7.0.2/Data/Configureeventlinebreaking

I would suggest, for better indexing performance, to explicitly define the event parsing rules for your sourcetypes, which is done on props.conf on Indexer or Intermediate/Heavy forwarders (Splunk Enterprise instance acting as forwarder) whichever comes first in the data flow. A sample configuration based on your sample events could be this (assuming you need current time as _time for your events as your events don't have timestamp in them)

[YourSourceTypeHere]
LINE_BREAKER = ([\r\n]+)(?=\[\w+)
SHOULD_LINEMERGE =false
DATETIME_CONFIG = CURRENT

[aa123s]

Apologies for not specifying it in the first place, but I did already test (and now retested) your suggestion without success.

[micahkemp]

Did you place the configuration on the search head, indexer, heavy forwarder, universal forwarder, or...?

[somesoni2]

And did you restart the splunk instance after making the change(assuming it was place at right instance)?

[aa123s]

I added the configuration into the local/props.conf of a universal forwarder. Said universal forwarder is the only one that writes to the particular sourcetype that I am using for testing.
The splunkd was duly restarted each time

[somesoni2]

The event parsing doesn't happen on UF (for non structured data), so you need to put these changes on your Indexer/Heavy Forwarders (wherever this UF is sending data to). Preferably create an app and put the props.conf under that app ($Splunk_Home/etc/apps/YourPropsApp/local/props.conf). A restart of Splunk would be required where you make the change. Also remember to use correct LINE_BREAKER regex (don't miss any escape characters).

[aa123s]

Unfortunately I don't have access to Indexer/HF... I will have to look for another solution, like breaking those lines at search time. Thanks anyways!

[micahkemp]

Youl should get the HF configured properly, even if it's not you who performs it. Correct data onboarding is critical, and event breaking is one of the major parts of onboarding.

You will be unhappy later if you try to do this at search time going forward.

[aa123s]

I believe this is how to dump my running configuration. (My stanza is named unicorn:legacy)

./bin/splunk cmd btool props list unicorn
[unicorn:legacy]
ANNOTATE_PUNCT = True
AUTO_KV_JSON = true
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = False
CHARSET = UTF-8
DATETIME_CONFIG = current
HEADER_MODE =
LEARN_SOURCETYPE = true
LINE_BREAKER = ([\r\n]+)(?=[\w+)
LINE_BREAKER_LOOKBEHIND = 100
MAX_DAYS_AGO = 2000
MAX_DAYS_HENCE = 2
MAX_DIFF_SECS_AGO = 3600
MAX_DIFF_SECS_HENCE = 604800
MAX_EVENTS = 256
MAX_TIMESTAMP_LOOKAHEAD = 128
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner
SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
SHOULD_LINEMERGE = False
TRANSFORMS =
TRUNCATE = 10000
detect_trailing_nulls = false
maxDist = 100
priority =
sourcetype =