Splunk Search

How to configure line breaking for mixed single line and multiline logs?

jamesvz84
Communicator

Given the following log format, is it possible to store the consecutive GROUPED/GROUPED_DET lines into one event while leaving LOGTYPE1 & LOGTYPE2 lines as singled-lined events? One of my main issues is also trying to come up with a regex for the LINE_BREAKER that will ONLY break if the line does NOT contain GROUPED : or GROUPED_DET:. Seems like Splunk will only look for a positive condition (break if x exists) rather than a negative condition (break if line does NOT contain x). Is that the case, or is there a regex or other settings that will allow proper breaking on this log?

LOGTYPE1 :2015/02/28 18:13:04.253|2015/02/28|118:13:04.252|6|LOC|4|0.01||||||||||||||0|50|41.25
LOGTYPE1 :2015/02/28 18:13:26.002|2015/02/28|18:13:26.002|5|GPS|3|1|39:48.0219|N|105:4.0201|W|2|10|1.0
GROUPED :2015/02/28 18:13:57.591:GROUP:Rcvd Office msg [1080] from AG
GROUPED_DET:2015/02/28 18:13:57.591:GROUP:000 | 04 04 38 01 11 00 00 09 00 03 d1 75 54 f2 05 66 | ..8........uT..f
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:001 | 28 00 37 00 00 62 6e 73 66 2e 62 3a 70 74 63 73 | (.7..xxxx.b:ptcs
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:002 | 72 76 72 00 62 6e 73 66 2e 6c 2e 62 6e 73 66 2e | rvr.xxxx.l.xxxx.
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:003 | 34 34 31 37 3a 69 74 63 00 42 4e 53 46 01 00 00 | 4417:itc.XXXX...
GROUPED_DET:2015/02/28 18:13:57.592:GROUP:004 | 00 00 7d a3 6b f9                               | ..}.k.
LOGTYPE2 :2015/02/28 18:13:57.595|2015/02/28|18:13:57.594|21|EMP|2|1080|1|1|04043801110000090003d17554f205662800370000626e73662e623a7074637372767200626e73662e6c2e626e73662e343431373a69746300424e534601000000007da36bf9
GROUPED :2015/02/28 18:13:58.546:GROUP:Sending Office msg [2080]
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:000 | 04 08 20 04 11 00 00 3b 00 03 d1 75 54 f2 05 66 | .. ....;...uT..f
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:001 | 28 01 2c 00 20 62 6e 73 66 2e 6c 2e 62 6e 73 66 | (.,. xxxx.l.xxxx
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:002 | 2e 34 34 31 37 3a 69 74 63 00 62 6e 73 66 2e 62 | .4417:itc.xxxx.b
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:003 | 3a 70 74 63 73 72 76 72 00 42 4e 53 46 00 00 00 | :ptcsrvr.XXXX...
GROUPED_DET:2015/02/28 18:13:58.546:GROUP:004 | 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:005 | 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 00 f8 | ................
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:006 | 65 2e 0d e3 c0 35 12 18 36 21 7b 02 00 02 04 01 | e....5..6!{.....
GROUPED_DET:2015/02/28 18:13:58.547:GROUP:007 | 00 00 00 00 cc da af de                         | ........
LOGTYPE1 :2015/02/28 18:13:58.550|2015/02/28|18:13:58.550|21|EMP|2|2080|1|1|040820041100003b0003d17554f2056628012c0020626e73662e6c2e626e73662e343431373a69746300626e73662e623a7074637372767200424e53460000000000000000000000000000000000000000000000000000000000000000a400f8652e0de3c035121836217b
0 Karma
1 Solution

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)

View solution in original post

jamesvz84
Communicator

The following worked (uses negative lookahead regex):

[sourcetype]
SHOULD_LINEMERGE = true 
LINE_BREAKER = ([\r\n]+)(?!GROUPED_DET)
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...

Updated Data Management and AWS GDI Inventory in Splunk Observability

We’re making some changes to Data Management and Infrastructure Inventory for AWS. The Data Management page, ...