Getting Data In

Line_Breaker question

PaulEscher
Explorer

[operlog]

LINE_BREAKER = (?m)(.\d{7}.\d\d:\d\d:\d\d.\d\d)

SHOULD_LINEMERGE = false

Why do my events have the text that I specified in my line_breaker removed?

Are my parens wrong? Should I add a different command?

In a nutshell I want my line break to happen when the weird date format shows up. But I want the date format to be in the event.

Thanks,
Paul

Tags (1)
1 Solution

lguinn2
Legend

One way to think about this is: LINE_BREAKER "defines" the line-break characters. The "line breaks" (defined by the regex capture group) are removed. This is the correct behavior for the LINE_BREAKER. See the Splunk docs on line breaking

I think that you probably want BREAK_ONLY_BEFORE

 BREAK_ONLY_BEFORE = (.\d{7}.\d\d:\d\d:\d\d.\d\d)

or maybe MUST_BREAK_AFTER

View solution in original post

dart
Splunk Employee
Splunk Employee

When using LINE_BREAKER you have a regular expression in up to three parts:

LINE_BREAKER = the previous event end (the data between events) the new event here

That is to say only the part in the capturing group is removed. So to break only on newlines followed by digits as per your pattern:
LINE_BREAKER = ([\r\n]+).\d{7}.\d\d:\d\d:\d\d.\d\d

This seems to cause some confusion, but using LINE_BREAKER (with SHOULD_LINEMERGE = false) is my preferred method as it only requires remembering one thing, and covers most cases in a quick and simple way.

m2oswald
Explorer

Thank you - this was exactly what I was trying to figure out.  I had a regex and couldn't figure out why only part of it was disappearing.  I didn't understand about the capture group.

0 Karma

fk319
Builder

the 'd' for digit does not seem correct to me, try '\d'

0 Karma

fk319
Builder

when I posted it there was a slash before the d, so try '\d'

0 Karma

lguinn2
Legend

One way to think about this is: LINE_BREAKER "defines" the line-break characters. The "line breaks" (defined by the regex capture group) are removed. This is the correct behavior for the LINE_BREAKER. See the Splunk docs on line breaking

I think that you probably want BREAK_ONLY_BEFORE

 BREAK_ONLY_BEFORE = (.\d{7}.\d\d:\d\d:\d\d.\d\d)

or maybe MUST_BREAK_AFTER

lguinn2
Legend

Further, BREAK_ONLY_BEFORE (and MUST_BREAK_AFTER) only require that you supply a string that uniquely appears in the first line (or last line) of the event - the regular expression is unanchored.

LINE_BREAKER requires a regular expression that is anchored both at the end of the last event line and the beginning of the first event line.

It may be faster to use LINE_BREAKER, but what good is that if the regular expression is wrong?

0 Karma

lguinn2
Legend

You are right that Mike's comment is correct and I was unclear. The capture portion of the regular expression is the only part that is removed when you use LINE_BREAKER.

Perhaps LINE_BREAKER is preferred for people who know regular expressions. In my experience, the number of people who can write a proper LINE_BREAKER regular expression is quite small. The manual actually says that LINE_BREAKER "might increase your indexing speed, but is somewhat more difficult to work with."

0 Karma

msettipane
Splunk Employee
Splunk Employee

LINE_BREAKER is the preferred method. Please see Mike's post below.

PaulEscher
Explorer

Thanks lguinn! I didn't see the relationship to the \r\n and the removal of them to create an event. Once I saw that the MUST_BREAK_AFTER made perfect sense.

Thanks again.
Paul

0 Karma

lguinn2
Legend

Thanks Lowell - I have edited my answer to correct the missing backslashes!

0 Karma

Lowell
Super Champion

Paul, note that the backslashes before the "d"s were removed. The text formatting is a bit messed up unfortunately. 😞 The suggestion here is right on!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...