Splunk Search

Regex to Select Data Between Line Breaks

dfrench151
Explorer

Hello,

I am trying to create a regex so that I can have all data in between line breaks as one event. Here is a sample of the data I'm working with:

isDraggingObject   : True
id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition          : {or, matches System.Object[] (?i)(^.*?host failure 
                     alert.*?www\.jennycraig\.com\.au.*?$), matches System.Object[] 
                     (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {route PVG22KK, severity warning}

isDraggingObject   : True
id                 : 3b5aa785-b854-4e43-900a-225da5786a27
condition          : {or, matches System.Object[] 
                     (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PVG22KK}

**isDraggingObject   : True
id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), 
                     matches System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PW0VV83}**

The aim is to get all data as one event.

Thanks in advance for you help. I've been trying multiple different regex expressions, but just can't figure it out...

0 Karma

woodcock
Esteemed Legend

Your description is strange to me. If you mean that there is a blank line between events, then this will do it:

[<YourSourcetypeHere>]
SHOULD_LINEMERGE = false
LINEBREAKER = ([\r\n]\s*[\r\n]+)
0 Karma

robinettdonWY
Path Finder

Assuming you want everything in between the empty lines in 1 capture group (named "event")... this should work.

edit: sorry for cut and paste silliness.
(?<event>^.\S*[^\n<]*(?:(?:<(?!)|\n(?!$))[^\n]*)*(?:|\n$|\z))

Demo

0 Karma

dfrench151
Explorer

So would this be in the props.conf file?
LINE_BREAKER = (?^.\S*^\n<*(?:|\n$|\z))

0 Karma

robinettdonWY
Path Finder

look at my last edit.... i think you need to name the capture group (my example I named it "event" ).

https://regex101.com/r/aRu6NA/1

0 Karma

dfrench151
Explorer

I tried that example as well and for some reason it is still combining all the data into one event.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Are you saying this data currently in multiple lines and you want to combine it into a single line? Do you want to do this at index time or search time?

---
If this reply helps you, Karma would be appreciated.
0 Karma

dfrench151
Explorer

I want this to be completed at the time I index the date. Even 1 should be:
isDraggingObject : True
id : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition : {or, matches System.Object(^.?host failure
alert.
?www.jennycraig.com.au.?$), matches System.Object(^.?\bwarning\b.?www.jennycraig.com.au.?$)}
catch_all : False
advanced_condition : {}
actions : {route PVG22KK, severity warning}

Event 2:
isDraggingObject : True
id : 3b5aa785-b854-4e43-900a-225da5786a27
condition : {or, matches System.Object(^.?\bcritical\b.?www.jennycraig.com.au.*?$)}
catch_all : False
advanced_condition : {}
actions : {severity critical, route PVG22KK}

Currently splunk is just grouping everything together into one event.

0 Karma

dmarling
Builder

Is this logging format some kind of application standard or is this something that someone made in house. I ask because the way it's currently formatted makes it difficult for index time field extractions. I have a search time extraction that can be used to accomplish what I believe you are trying to do (create key value pairs) using a run anywhere example with your data that was provided on your question:

| makeresults count=3 
| streamstats count as counter 
| eval _raw=case(counter=1, " isDraggingObject   : True
 id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
 condition          : {or, matches System.Object[] (?i)(^.*?host failure 
                      alert.*?www\.jennycraig\.com\.au.*?$), matches System.Object[] 
                      (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
 catch_all          : False
 advanced_condition : {}
 actions            : {route PVG22KK, severity warning}", counter=2, " isDraggingObject   : True
 id                 : 3b5aa785-b854-4e43-900a-225da5786a27
 condition          : {or, matches System.Object[] 
                      (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
 catch_all          : False
 advanced_condition : {}
 actions            : {severity critical, route PVG22KK}", counter=3, " **isDraggingObject   : True
 id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
 condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), 
                      matches System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
 catch_all          : False
 advanced_condition : {}
 actions            : {severity critical, route PW0VV83}**") 
| fields - counter _time 
| rex field=_raw mode=sed "s/(\n|^)\s+(\w+\s+):/█\1\2:/g" 
| makemv _raw delim="█" 
| rex mode=sed field=_raw "s/█//g"
| rex mode=sed field=_raw "s/\n/ /g"
| extract kvdelim=":" pairdelim="
"

The first sed statement is placing a unique character to be inserted into the event, which we can then use to make it a multi-valued results that splits each key value pairing into a unique value of the raw data. It then removes the unique character and then removes all line breaks and makes them spaces. After that is done you can run the extract command and it will produce proper key value pairing.

If this is a custom generated log event, I would suggest that they consider making it key value format by wrapping their values with quotes and changing the colon to an equal sign or follow the Windows events formatting standard if you want line breaks without quotes in your values.

If this comment/answer was helpful, please up vote it. Thank you.
0 Karma

dfrench151
Explorer

The document is created from a powershell script that pulls data through an API connection with an online application. When the information is indexed, there are not line breaks...:

isDraggingObject   : True
id                 : afbbdeb7-9fd4-4b53-ab17-742809154ba9
condition          : {or, matches System.Object[] (?i)(^.*?host failure alert.*?www\.jennycraig\.com\.au.*?$), matches 
                     System.Object[] (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {route PVG22KK, severity warning}
isDraggingObject   : True
id                 : 3b5aa785-b854-4e43-900a-225da5786a27
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com\.au.*?$)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PVG22KK}
isDraggingObject   : True
id                 : a8420998-fbca-486b-9ff7-d03b9e16536e
condition          : {or, matches System.Object[] (?i)(^.*?\bcritical\b.*?www\.jennycraig\.com$), matches 
                     System.Object[] (?i)(^.*?\bcritical\b.*?locations\.jennycraig\.com)}
catch_all          : False
advanced_condition : {}
actions            : {severity critical, route PW0VV83}
isDraggingObject   : True
id                 : d9837013-68c9-42bf-a91f-8dd8a94ca377
condition          : {or, matches System.Object[] (?i)(^.*?host failure alert.*?www\.jennycraig\.com$), matches 
                     System.Object[] (?i)(^.*?\bwarning\b.*?www\.jennycraig\.com$), matches System.Object[] 
                     (?i)(^.*?host failure alert.*?locations\.jennycraig\.com)...}
catch_all          : False
advanced_condition : {}
actions            : {route PW0VV83, severity warning}
isDraggingObject   : True
id                 : 20fc7d82-d17d-443e-9802-c8f2df462ce9
condition          : {or, equals System.Object[] critical: Page Failure alert on 
                     https://uw2pobi11.sonic.jennycraig.com:9503/analytics, equals System.Object[] critical: Page 
                     Failure alert on https://uw2pobi11.sonic.jennycraig.com:9501/console, equals System.Object[] 
                     critical: Page Failure alert on https://uw2pobi11.sonic.jennycraig.com:9501/em...}
catch_all          : False
advanced_condition : {}
actions            : {route PO77HX2}

I tried out your code in one of the searches and it produced no results.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...