I need a little guidance on rex field extraction on the following "redacted" security log. Unfortunately, I don't have permission to use field extraction gui so need to rex in the search.
2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error], [pid 12345:tid 1234567890], [client 192.168.1.1], Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. [file "some_attack.conf"], [line "111"], [id "111"], [rev "1"], [msg "Some Attack: blah blah Detected."], [data "Matched Data: blah blah ARGS:file: <foo=bar"], [severity "CRITICAL"], [ver "foo/2.2. [hostname "bar.com"], [uri "/foo/bar/foobar"], [unique_id "a1b2c3-a1b2c3"],
As you can see, there are commas and [ brackets] delimiting the log and splunk is extracting some fields like the time field correctly, but I would like to extract more fields
like this
2018:02:09-23:04:30
blahblah httpd[11111], : <-----------field 1
[foo-security:error], <-----------field 2
[pid 12345:tid 1234567890], <-----------field 3
[client 192.168.1.1], <-----------field 4
Foo-Security: Warning. Pattern match "some expression..." at ARGS:file. <-----------field 5
[file "some_attack.conf"], <-----------field 6
[line "111"], <-----------field 7
[id "111"], <-----------field 8
[rev "1"], <-----------field 9
[msg "Some Attack: blah blah Detected."], <-----------field 10
[data "Matched Data: blah blah ARGS:file: <foo=bar"], <-----------field 11
[severity "CRITICAL"], <-----------field 12
[ver "foo/2.2. [hostname "bar.com"], <-----------field 13
[uri "/foo/bar/foobar"], <-----------field 14
[unique_id "a1b2c3-a1b2c3"], <-----------field 15
I am a bit rusty on this so any help is much appreciated.
Thank you
Hi,
Try like this run anywhere search:
|makeresults|eval _raw="2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error], [pid 12345:tid 1234567890], [client 192.168.1.1], Foo-Security: Warning. Pattern match \"some expression...\" at ARGS:file. [file \"some_attack.conf\"], [line \"111\"], [id \"111\"], [rev \"1\"], [msg \"Some Attack: blah blah Detected.\"], [data \"Matched Data: blah blah ARGS:file: <foo=bar\"], [severity \"CRITICAL\"], [ver \"foo/2.2. [hostname \"bar.com\"], [uri \"/foo/bar/foobar\"], [unique_id \"a1b2c3-a1b2c3\"], "|rex "^[\d\:]+-[\d\:]+(?<_raw>.*)"|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
Hi,
Try like this run anywhere search:
|makeresults|eval _raw="2018:02:09-23:04:30 blahblah httpd[11111], : [foo-security:error], [pid 12345:tid 1234567890], [client 192.168.1.1], Foo-Security: Warning. Pattern match \"some expression...\" at ARGS:file. [file \"some_attack.conf\"], [line \"111\"], [id \"111\"], [rev \"1\"], [msg \"Some Attack: blah blah Detected.\"], [data \"Matched Data: blah blah ARGS:file: <foo=bar\"], [severity \"CRITICAL\"], [ver \"foo/2.2. [hostname \"bar.com\"], [uri \"/foo/bar/foobar\"], [unique_id \"a1b2c3-a1b2c3\"], "|rex "^[\d\:]+-[\d\:]+(?<_raw>.*)"|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
Thank you for the reply. Maybe I am misunderstanding or I did not explain clearly.
I was looking for a way that will eval all the log events in _raw, without having to copy and paste like above.
The example I provided is the un-parsed / un-extracted raw text I need to rex into usable fields. Does that make sense...?
if the event is already indexed then have you tried this:
<base_search>|makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
Here in <base search>
include the query from where you are getting events/raw text like index=indexname
use like this:
index=A |makemv _raw delim=","|eval field0=mvindex(_raw,0),field1=mvindex(_raw,1),field2=mvindex(_raw,2),field3=mvindex(_raw,3),field4=mvindex(_raw,4),field5=mvindex(_raw,5),field6=mvindex(_raw,6),field7=mvindex(_raw,7)
the logs are in an index=A and sourcetype = A, etc... is there a way to use your method with a specific index and sourcetype?
for msg field number 10 would the following spl code be correct?
... | regex _raw="msg+.[^],]*