Splunk Search

Multi Line field extraction

santorof
Path Finder

Trying to do an expression that would extract IP's that are below the Client IP: line. Im looking to pull out each IP that is separated by a comma individually. In my raw event log there is a blank line above the Client IP line and also below the line that gives the IP addresses. Its possible I can have more than one or two IP's per log.

Client IP:
0.0.0.0,0.0.0.0

I also have another field in the same log that gives a username. There is a blank line above the error message and below the username. I was looking to extract everything after the : and before the -The user name.
I tried UserName="Error message:(/w)-The user name or password is incorrect"

Error message:
username@test.com-The user name or password is incorrect

0 Karma

woodcock
Esteemed Legend

Like this:

... | rex "(?ms)Client\s+IP:[\r\n\s]*(?<ClientIPs>[\d\.\,]+)" | rex "(?ms)Error\s+message:[\r\n\s]*(?<EmailAddress>[^\s\-]+)"
0 Karma

adayton20
Contributor

From time to time I run into an issue extracting fields from _raw with Windows logs containing new lines, but sometimes what I'm looking for is in the Message field bunched together too. If that's the case for you. Give this a try:

 | rex field=Message "(IP\:\s+|\d+\,)(?P<ClientIP>.[^\s]*)"
 | makemv delim="," ClientIP 

This should extract the IPs and separate them into their own values. If you don't want the IPs as their own values, remove the makemv portion of it.

Also, someone posted a log the other day from an ADFS log where the Client IP had a . at the end of the second IP. If that appears in your events, give this a try:

 | rex field=Message "(IP\:\s+|\d+\,)(?P<ClientIP>.*)."
 | makemv delim="," ClientIP 

For the username, give one of these a try:

| rex field=Message "message\:\s(?P<username>.*)-"

or

  | rex field=Message "message\:\s(?P<username>.[\s]*)"

or

 | rex field=Message "message\:\s(?P<username>.[\-]*)"

If none of that works, would you mind providing a sanitized sample of the entire log?

0 Karma

niketn
Legend

Are using trying the rex command or IFX?
In either case you can turn on Access Flag (?s), due to the same \s+ will match newline characters also. Following is an example:

rex field=_raw "(?s)Error message:\s+(?<Username>.*)-The user name or password is incorrect" 

Both rex and IFX worked for me... If Rex is not working for you please try Interactive Field Extraction in Splunk and let Splunk come up with required regular expression. Following is the regular expression IFX came up with ^(?:[^ \n]* ){4}\s+(?P<Username>[^\-]+). You can also try this instead in rex to test.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

santorof
Path Finder

Just tried running this to extract but did not get any results. The \s+ command is telling my extraction to match one or more times to whitespace which means go from one line to the next line until you hit your first character thats not whitespace correct?

0 Karma

niketn
Legend

Yes... I tried to attach image to my answer since both rex as well as IFX worked for me. However, I am unable to do so... I have added regular expression that IFX came up with to my answer. Please test that in rex. If not Try Field Extraction yourself.

One of the other option would be to remove newline character from your _raw events using eval with replace, but that will be too expensive.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

santorof
Path Finder

Not sure whats going on with my extract. I tried the below and got results the first time through extract preview with just the username@test.com. I put in the extraction for all users and when going back in the search the field wasent there. I deleted the extract and readded it and its not finding results again.

^(?:[^ \n]* ){4}\s+(?P[^-]+)

0 Karma

niketn
Legend

Using Extract fields method in your Splunk Search you can create Field Extraction yourself using Regular Expression on your data. Make sure there are no non-matches in your selected events or you would need to adjust the regular expression by including non-matched events.

Follow the documentation to come up with your field extraction: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/ExtractfieldsinteractivelywithIFX

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...