Getting Data In

Pull out fields embedded in logs

mdavis43
Path Finder

I'm looking for information about how to pull out field information from inside the log messages. For example...

Message=(Error) I/O error on file system 'prodops' operation WRITE inode (Message repeated 4732 times)

Message=(Error) I/O error on file system 'proxy' operation WRITE inode (Message repeated 4 times)

Message=(Error) I/O error on file system 'wwtowip' operation WRITE inode

These come from Windows event logs and I want to be able to sort on how many times these errors happen to each individual filesystem.

0 Karma
1 Solution

lguinn2
Legend

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system 

I've probably made some typo in the regular expression... but I hope you get the idea

View solution in original post

lguinn2
Legend

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system 

I've probably made some typo in the regular expression... but I hope you get the idea

mdavis43
Path Finder

Thanks, this looks great for long term usage.

0 Karma

lukejadamec
Super Champion

You should use rex to create a field from within the Message field that you can search on later.

index=main sourcetype="*security*" | rex field=Message "... error on file system '(?<filesystem>.*)' | stats count by filesystem
0 Karma

mdavis43
Path Finder

Thanks! This worked great inline.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...