Splunk Search

How can I do field extraction on a security log using regex?

Log_wrangler
Builder

I need a little guidance on rex field extraction on the following "redacted" security log. Unfortunately, I don't have permission to use field extraction gui so need to rex in the search.

2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error],  [pid 12345:tid 1234567890],  [client 192.168.1.1],  Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. [file "some_attack.conf"],  [line "111"],  [id "111"],  [rev "1"],  [msg "Some Attack: blah blah Detected."],  [data "Matched Data: blah blah ARGS:file: <foo=bar"],  [severity "CRITICAL"],  [ver "foo/2.2. [hostname "bar.com"],  [uri "/foo/bar/foobar"],  [unique_id "a1b2c3-a1b2c3"], 

As you can see, there are commas and [ brackets] delimiting the log and splunk is extracting some fields like the time field correctly, but I would like to extract more fields

like this

2018:02:09-23:04:30 
blahblah httpd[11111], : <-----------field 1
[foo-security:error],  <-----------field 2 
[pid 12345:tid 1234567890],  <-----------field 3
[client 192.168.1.1],  <-----------field 4
Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. <-----------field 5
[file "some_attack.conf"],  <-----------field 6
[line "111"],  <-----------field 7
[id "111"],  <-----------field 8
[rev "1"],  <-----------field 9
[msg "Some Attack: blah blah Detected."],  <-----------field 10
[data "Matched Data: blah blah ARGS:file: <foo=bar"],  <-----------field 11
[severity "CRITICAL"],  <-----------field 12
[ver "foo/2.2. [hostname "bar.com"],  <-----------field 13
[uri "/foo/bar/foobar"],  <-----------field 14
[unique_id "a1b2c3-a1b2c3"], <-----------field 15

I am a bit rusty on this so any help is much appreciated.

Thank you

0 Karma
1 Solution

493669
Super Champion

Hi,
Try like this run anywhere search:

|makeresults|eval _raw="2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error],  [pid 12345:tid 1234567890],  [client 192.168.1.1],  Foo-Security: Warning. Pattern match \"some expression...\" at ARGS:file. [file \"some_attack.conf\"],  [line \"111\"],  [id \"111\"],  [rev \"1\"],  [msg \"Some Attack: blah blah Detected.\"],  [data \"Matched Data: blah blah ARGS:file: <foo=bar\"],  [severity \"CRITICAL\"],  [ver \"foo/2.2. [hostname \"bar.com\"],  [uri \"/foo/bar/foobar\"],  [unique_id \"a1b2c3-a1b2c3\"], "|rex "^[\d\:]+-[\d\:]+(?<_raw>.*)"|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)

View solution in original post

0 Karma

493669
Super Champion

Hi,
Try like this run anywhere search:

|makeresults|eval _raw="2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error],  [pid 12345:tid 1234567890],  [client 192.168.1.1],  Foo-Security: Warning. Pattern match \"some expression...\" at ARGS:file. [file \"some_attack.conf\"],  [line \"111\"],  [id \"111\"],  [rev \"1\"],  [msg \"Some Attack: blah blah Detected.\"],  [data \"Matched Data: blah blah ARGS:file: <foo=bar\"],  [severity \"CRITICAL\"],  [ver \"foo/2.2. [hostname \"bar.com\"],  [uri \"/foo/bar/foobar\"],  [unique_id \"a1b2c3-a1b2c3\"], "|rex "^[\d\:]+-[\d\:]+(?<_raw>.*)"|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
0 Karma

Log_wrangler
Builder

Thank you for the reply. Maybe I am misunderstanding or I did not explain clearly.

I was looking for a way that will eval all the log events in _raw, without having to copy and paste like above.

The example I provided is the un-parsed / un-extracted raw text I need to rex into usable fields. Does that make sense...?

0 Karma

493669
Super Champion

if the event is already indexed then have you tried this:

<base_search>|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)

Here in <base search> include the query from where you are getting events/raw text like index=indexname

0 Karma

493669
Super Champion

use like this:

index=A |makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
0 Karma

Log_wrangler
Builder

the logs are in an index=A and sourcetype = A, etc... is there a way to use your method with a specific index and sourcetype?

0 Karma

Log_wrangler
Builder

for msg field number 10 would the following spl code be correct?

... | regex _raw="msg+.[^],]*     
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...