Splunk Search

Field extraction for Log File Entries with Pipe delimiters

mbasharat
Contributor

Hi,

I have a log file I am monitoring. Log file entries have pipe delimited field entries as below:

LE Variation 1:
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|... SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|

LE Variation 2:
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|...

Both variations exist in the log and I need both. The only differences among the two for distinction is that |success| defines successful transaction and anything other than |success| is a failure.

I need fields to be extracted using regex or eval in Splunk search please. You can rename them as samples and I will update at my end as needed.

Thanks in-advance.

0 Karma
1 Solution

to4kawa
Ultra Champion
| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

View solution in original post

0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="
[default task-2] 2020-01-24 13:10:54,598 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|REQSTI003000004:Invalid SOAP message format,Invalid SOAP message format: abc-def.5.2.2.2.2: The value '10.1' of element 'ns1:WSDLVersionNum' does not match the {value constraint} value '10.3'.|
[default task-11] 2020-01-23 12:45:01,851 INFO sample.sample.sample.sample.sample.sample.StatLogger - ABCStat|XYZ|11111111111111111111|http://www.abc.com/XYZ/123/ABCD/submission|2020-01-24T13:10:52.414Z|2020-01-24T13:10:54.595Z|2181|0|3909|success|"
| makemv delim="
" _raw 
| stats count by _raw
| table _raw
`comment("this is your sample")`
| eval tmp=mvzip(split(_raw,"|"),mvrange(0,mvcount(split(_raw,"|"))))
| eval success=mvindex(split(_raw,"|"),9)

Hi, @mbasharat
Fields are left for clarity.

0 Karma

mbasharat
Contributor

Slight adjustment based on my use case and worked great. Thanks!!!

0 Karma

mydog8it
Builder

I think this will work for you...

Your_search | rex  "[|:](?P<Status>\D\w{6,8})[| ]" | stats count by Status

The caveat is the 'failure' messages would need to be non-digit and between 6&8 characters in length (otherwise adjust the regex). Careful modifying the length too much or it may introduce false positives.

0 Karma

mbasharat
Contributor

Hi mydog8it,

First, I need to have the pipe delimited fields extracted in SPL search. For example:

task: [default task-11]
timestamp: 2020-01-23 12:45:01,851
loglevel: INFO
URL: sample.sample.sample.sample.sample.sample.StatLogger -
type: ABCStat
app: XYZ
id: 11111111111111111111
surl: http://www.abc.com/XYZ/123/ABCD/submission
timestamp2: 2020-01-24T13:10:52.414Z
timestamp3: 2020-01-24T13:10:54.595Z
transactionid: 2181
code: 0
submission: 3909
status: success

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What have you tried so far?

---
If this reply helps you, Karma would be appreciated.
0 Karma

mbasharat
Contributor

I tried using field extractor but that will not do it because it should be done at the main config level. While that piece is being discussed, I need to go ahead and do search time extraction so I can create reports as needed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...