Splunk Search

How to extract specific lines in a multiline event based on regex match?

popdeluxe
New Member

I am trying to analyze exception logging that is written across multiple lines, and extract only certain lines of the event into fields. I have been reading documentation and posts which seem to suggest defining stanzas in transforms.conf and props.conf would be the preferred way to tackle this. I have tried to implement to no avail and am lost! I would appreciate any ideas/suggestions on how to properly implement!!

Here are a few example log snippets to help demonstrate the challenge. Given the following, I need to extract...

(a) exception message (in bold/italics)
(b) the first calling method from either SOURCE_B or SOURCE_C, but not SOURCE_A

(1)
20160825 12:51:16 unhandled error from dispatcher, sender:System.Windows.Threading.Dispatcher
System.NullReferenceException: Object reference not set to an instance of an object.
at SOURCE_A.Method(Object sender, ExecutedRoutedEventArgs e)
at SOURCE_A.Method(Object sender, ExecutedRoutedEventArgs e)
at SOURCE_B.Method(Object sender, ExecutedRoutedEventArgs e, CommandBinding commandBinding)

(2)
20160825 12:53:16 unhandled error from dispatcher, sender:System.Windows.Threading.Dispatcher
System.Runtime.InteropServices.COMException ().
at SOURCE_C.Method(FORMATETC& format, STGMEDIUM& medium)

with the following results

(1)
UE_msg: System.NullReferenceException: Object reference not set to an instance of an object.
UE_method: SOURCE_B.Method(Object sender, ExecutedRoutedEventArgs e, CommandBinding commandBinding)

(2)
UE_msg: System.Runtime.InteropServices.COMException ()
UE_method: SOURCE_C.Method(FORMATETC& format, STGMEDIUM& medium)

The logging is not very structured...but all of these exceptions include the "unhandled error" string pattern, with the high-level "exception message" following on the next line that I need to extract, then SOURCE_B or SOURCE_C methods following below that somewhere in the stacktrace. So my thoughts are to define a REGEX stanza in transforms.config as follows

(transforms.config)

[UE_regex]
REGEX = (?m)(unhandled error.\*\n)(.\*\\.)((SOURCE_B|SOURCE_C).*\\))
FORMAT = UE_msg::$2  UE_method::$3

(props.config)

[UE]
REPORT-UE = UE_regex

lastly...try to table results....

source="c:\\logs\\perf*" sourcetype="UE" | table results

I have been tweaking the REGEX patterns, groupings, tried ditching transforms.config and tried defining just an EXTRACT in props.config. But nothing has yielded any results. At this point I can't tell if I'm even on the correct path anymore and would appreciate some guidance!

thanks!!!

0 Karma
1 Solution

sundareshr
Legend

Try separate REPORT. Like this

*props*

[UE]
REPORT-UE = UE_msg
REPORT-UE = UE_method

*transforms*

[UE_msg]
REGEX = (?ms)unhandled error.*\n(?<UE_msg>[^\n]+)

[UE_method]
REGEX = at\s+(?<UE_method>SOURCE_[B|C])

View solution in original post

0 Karma

sundareshr
Legend

Try separate REPORT. Like this

*props*

[UE]
REPORT-UE = UE_msg
REPORT-UE = UE_method

*transforms*

[UE_msg]
REGEX = (?ms)unhandled error.*\n(?<UE_msg>[^\n]+)

[UE_method]
REGEX = at\s+(?<UE_method>SOURCE_[B|C])
0 Karma

popdeluxe
New Member

thank you sundareshr - this got me on the right track.

I tried the separate report, but it was only taking the last-assign stanza. i.e. UE_method was extracted but not UE_msg.

Continuing with this approach however, the following seems to work:

props.conf

REPORT-UEmsg = UE_msg
REPORT-UEmethod = UE_method

transforms.conf

[UE_msg]
REGEX = (?m)unhandled error.*(?System\D+\:.*)\.\s

[UE_method]
REGEX = (?m)unhandled error.*(?(SOURCE_B|SOURCE_C)\D+)\s

this has gotten me very close to what I need. I still am wrestling with the regex of the UE_method as it is extracting the rest of the stack trace instead of the specific line I want, but I will other posts for a solution and post a new one if needed.

thanks!

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...