All Apps and Add-ons

Unable to extract optional fields in splunk through regex

luv
Explorer

Hi, I have logs some what like this

......[ABC] - [YUP1,AConsole]
......[PQR] - [YUP1,PConsole]
......[ZAD] - [YUP1,DConsole]
......[SID] - [TYU3,2013-08-29,QConsole]
......[POP] - [TYU3,2013-08-30,TConsole]
......[IOL] - [TYU3,2013-09-01,XConsole]
......[DSW] - [GKFO,2013-09-12,iConsole,Payment1]
......[ESD] - [IOSD,2013-09-13,iConsole,Payment2]
......[ABC] - [YUP1,AConsole]
......[RTS] - [YUP1,RConsole]
......[SID] - [TYU3,2013-09-26,QConsole]
......[DSW] - [GKFO,2013-10-29,iConsole,Payment3]
......[EDS] - [EDC1,FConsole]

In the square brackets [(Field1),(Field2),(Field3),(Field4)] some of the fields are optional, they come and go in some events.

1st combination [(Field1),(Field3)]
2nd combination [(Field1),(Field2),(Field3)]
3rd combination [(Field1),(Field2),(Field3),(Field4)]
This means Field1 & Field3 are always there in the events but Field2 & Field4 are optional, I want to extract all of them.

This is the regex which i have come up with, But seems like it is not working for my case

rex field=_raw "\] - \[(?< Field1 >[^,]+)(,)?(?< Field2 >[^,]+)?,(?< Field3 >[^,\]]+)(,)?(< Field4 >[^\]]+)?"
PS- I have added a space between "<" and ">" because it text box was escaping it.
Any help would be really appreciated

Thanks 🙂

0 Karma
1 Solution

dmaislin_splunk
Splunk Employee
Splunk Employee

I think this will also work:

.+\[(?<FIELD1>[^,]+)\,?(?<FIELD2>[0-9-]+)?\,(?<FIELD3>\w+)\,?(?<FIELD4>\S+)?\]

View solution in original post

d29priyanka
New Member

Hi I have events look like this:

DISKBSIZE,T0001,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
DISKBSIZE,T3729,8.0,4.0,0.0,7.4,4.0,4.0,4.0,0.0,10.8,0.0,0.0,6.0,0.0,4.0,6.0,0.0,10.5,0.0,8.0,4.0,4.0,0.0,8.7,4.0,8.4,5.7,4.0,9.3,0.0,5.5,0.0
DISKBSIZE,T3728,10.2,4.0,0.0,0.0,5.7,27.8,5.7,8.6
DISKBSIZE,T3729,0.0,4.0,11.3,0.0,0.0,10.8,0.0,6.0,0.0,5.3,4.0,0.0,11.1,0.0,4.0,6.0,0.0,5.6,0.0,13.9,0.0,4.0,5.3,17.1,0.0,9.3,0.0,10.0,5.4,6.7,4.0,13.2,0.0,8.0,0.0

Some fields come and go.Need a regex which extracts multiple fields.
Here is the regex which i came up with

|rex field=_raw "(?i)DISKBSIZE,(?P[^,]),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),(?P[^,]\d+),?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?,?(?P[^,]\d+)?"

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

I think this will also work:

.+\[(?<FIELD1>[^,]+)\,?(?<FIELD2>[0-9-]+)?\,(?<FIELD3>\w+)\,?(?<FIELD4>\S+)?\]

dmaislin_splunk
Splunk Employee
Splunk Employee

You probably don't need to escape the comma but it never hurts to be literal. I should have just escaped the last one too.

0 Karma

luv
Explorer

just out of curiosity whats this "\" for?? between ".+\[(?[^,]+)" and ","
same with "?(?[0-9-]+)?" and ","
is there something which needs to be escaped?

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Thanks much!

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

Please accept this answer by checking the check mark. Thanks!

0 Karma

luv
Explorer

that worked!!! thank you so much 🙂

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can use following. Assumption there are atleast 2 fields and they are separated by comma. You can add more eval statements if you have more fields.

..your search..| rex field=_raw ".*\[(?<fieldlist>[^\]]+)" |eval fieldlist=split(fieldlist,",") 
| eval fieldcount=mvcount(fieldlist) 
| eval field1=mvindex(fieldlist,0) 
| eval field2=case(fieldcount > 2,mvindex(fieldlist,1),1=1,"")
| eval field3=case(fieldcount > 2,mvindex(fieldlist,2),1=1,mvindex(fieldlist,1))
| eval field4=case(fieldcount > 3,mvindex(fieldlist,3),1=1,"")

luv
Explorer

This is really cool way to extract fields,it worked....Will definitely try more of this in future 🙂

0 Karma

dmaislin_splunk
Splunk Employee
Splunk Employee

You said: This means Field1 & Field3 are always there in the events but Field3 & Field4 are optional, I want to extract all of them.

Did you mean Field2 & Field4 are optional?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...