Splunk Search

How to configure line breaking for mixed single line and multiline logs?

jamesvz84
Communicator

Given the following log format, is it possible to store the consecutive GROUPED/GROUPED_DET lines into one event while leaving LOGTYPE1 & LOGTYPE2 lines as singled-lined events? One of my main issues is also trying to come up with a regex for the LINE_BREAKER that will ONLY break if the line does NOT contain GROUPED : or GROUPED_DET:. Seems like Splunk will only look for a positive condition (break if x exists) rather than a negative condition (break if line does NOT contain x). Is that the case, or is there a regex or other settings that will allow proper breaking on this log?

LOGTYPE1 :2015/02/28 18:13:04.253|2015/02/28|118:13:04.252|6|LOC|4|0.01||||||||||||||0|50|41.25
LOGTYPE1 :2015/02/28 18:13:26.002|2015/02/28|18:13:26.002|5|GPS|3|1|39:48.0219|N|105:4.0201|W|2|10|1.0
GROUPED :2015/02/28 18:13:57.591:GROUP:Rcvd Office msg [1080] from AG
GROUPED_DET:2015/02/28 18:13:57.591:GROUP:000 | 04 04 38 01 11 00 00 09 00 03 d1 75 54 f2 05 66 | ..8........uT..f
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:001 | 28 00 37 00 00 62 6e 73 66 2e 62 3a 70 74 63 73 | (.7..xxxx.b:ptcs
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:002 | 72 76 72 00 62 6e 73 66 2e 6c 2e 62 6e 73 66 2e | rvr.xxxx.l.xxxx.
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:003 | 34 34 31 37 3a 69 74 63 00 42 4e 53 46 01 00 00 | 4417:itc.XXXX...
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:004 | 00 00 7d a3 6b f9                               | ..}.k.
LOGTYPE2 :2015/02/28 18:13:57.595|2015/02/28|18:13:57.594|21|EMP|2|1080|1|1|04043801110000090003d17554f205662800370000626e73662e623a7074637372767200626e73662e6c2e626e73662e343431373a69746300424e534601000000007da36bf9
GROUPED :2015/02/28 18:13:58.546:GROUP:Sending Office msg [2080]
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:000 | 04 08 20 04 11 00 00 3b 00 03 d1 75 54 f2 05 66 | .. ....;...uT..f
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:001 | 28 01 2c 00 20 62 6e 73 66 2e 6c 2e 62 6e 73 66 | (.,. xxxx.l.xxxx
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:002 | 2e 34 34 31 37 3a 69 74 63 00 62 6e 73 66 2e 62 | .4417:itc.xxxx.b
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:003 | 3a 70 74 63 73 72 76 72 00 42 4e 53 46 00 00 00 | :ptcsrvr.XXXX...
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:004 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:005 | 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 00 f8 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:006 | 65 2e 0d e3 c0 35 12 18 36 21 7b 02 00 02 04 01 | e....5..6!{.....
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:007 | 00 00 00 00 cc da af de                         | ........
LOGTYPE1 :2015/02/28 18:13:58.550|2015/02/28|18:13:58.550|21|EMP|2|2080|1|1|040820041100003b0003d17554f2056628012c0020626e73662e6c2e626e73662e343431373a69746300626e73662e623a7074637372767200424e53460000000000000000000000000000000000000000000000000000000000000000a400f8652e0de3c035121836217b
0 Karma
1 Solution

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)

View solution in original post

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...