Getting Data In

Line break with multiple Linebreaker

ss026381
Communicator

I have a log file something like this

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 
PQL% bla bla bla bla
bla bla blabla bla bla
PQL% bla bla bla bla

I want to break it at PStart or PQL% but if PQL% is within PStart and PEnd keep it one event.

For the above example, I want to have 3 events as follow

Event 1

PStart  bla bla bla bla 
PQL% bla bla bla bla
PEnd bla bla bla bla 

Event 2

PQL% bla bla bla bla
bla bla blabla bla bla

Event 3

PQL% bla bla bla bla

The code I have is following which doesn't work properly

[sourcetype]
LINE_BREAKER = ([\r\n]+)(?:(?:PStart|(?:PQL%))) 
MUST_NOT_BREAK_BEFORE = ([\r\n]+)(?:PEnd) 
Tags (2)
1 Solution

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

View solution in original post

acharlieh
Influencer

With the sample as given this is what I come up with:

[sourcetype]
LINE_BREAKER=([\r\n]+)(?:PStart|PEnd|PQL%)
SHOULD_LINEMERGE=true
BREAK_ONLY_BEFORE=^PStart|^PQL%
MUST_NOT_BREAK_AFTER=^PStart
MUST_BREAK_AFTER=^PEnd

First during the line breaking stage, we break up into "lines" that start PStart, PEnd or PQL% (This is only necessary if a PEnd line may contain carrage returns or line feeds, otherwise you can get away with the default ([\r\n]+) here

Then during line merging, we only break events before a PStart or PQL% line... this gets us our rough separation of events... but then with the MUST_NOT_BREAK_AFTER / MUST_BREAK_AFTER combination, we can ensure that from PStart to PEnd is a single block. (Obviously does not support embedding, but works single level).

ss026381
Communicator

This works, thanks...

0 Karma

woodcock
Esteemed Legend

Be sure to click Accept to close this question and reward the author of this MOST TERRIBLY EXCELLENT ANSWER!!!!

0 Karma

ss026381
Communicator

Thank you guys, I accepted, special thanks to Charlie.

0 Karma

woodcock
Esteemed Legend

This is a tough one. I am curious to see if there is a splunk-based solution. I've got nothing because negative-lookbehinds must be fixed-length.

0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...