Splunk Search

How to sort different error strings in one log file?

Bergans
Engager

Hi,
I'm currently importing log-files into Splunk, to monitor the different kind of Errors that passes through the system that are monitored.
Up to now I've only searched for the string 'ERROR' in each log file. Since a log file may contain many different kind of errors, the result is that many kind of Errors are presented together. I would like to sort/group the different kind of errors in one diagram.

The search as today is as follows (for one of the log-files)

ERROR source="/home/logs/DataTransferService.log"

The result would then consist of many different ERROR messages, similar to these three (as an example):

2014-08-19 12:00:00,394 [pool-1-thread-1] ERROR - Unexpected error com.eMeter.PIPe.datatransferservice.exception.DTSRuntimeException:…

2014-08-19 11:20:01,815 [pool-1-thread-4] ERROR - Invalid date. java.lang.NullPointerException:…

2014-08-19 11:20:01,814 [pool-1-thread-4] ERROR - SDP lookup failed [null]: id to load is required for loading [Additional Information:…
Tags (3)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

Well, it's not entirely clear what you want to achieve. Sorting and grouping is usually performed on fields. Say that you want to group all errors on the type of error ("Unexpected error", "Invalid date" or "SDP lookup failed" in your example), you need to extract this part of the message as a field (let's call it errType) and whatever comes after it is extracted as errMsg. This can be done in config files or inline in the search query. We'll do the latter here.

ERROR source="/home/logs/DataTransferService.log" 
| rex "\sERROR\s-\s(?<errType>[^\.\r\n:]+)(?<errMsg>.*)"
| eval niceTime = strftime(_time, "%F %T")
| stats list(niceTime) list(errMsg) by errType

This should result in an output like

Unexpected Error        2014-06-20 11:22:23     com.eMeter.PIPe.datatransferblahblahblahb
                        2014-06-21 12:23:34     some.other.errormessage
Invalid date            2014-06-23 22:32:21     Blah Blah Blah error

and so forth. Maybe something like that is what you're looking for?

/K

View solution in original post

Bergans
Engager

No, the Errors may be identified by 'ERROR - '*

0 Karma

kristian_kolb
Ultra Champion

Well, it's not entirely clear what you want to achieve. Sorting and grouping is usually performed on fields. Say that you want to group all errors on the type of error ("Unexpected error", "Invalid date" or "SDP lookup failed" in your example), you need to extract this part of the message as a field (let's call it errType) and whatever comes after it is extracted as errMsg. This can be done in config files or inline in the search query. We'll do the latter here.

ERROR source="/home/logs/DataTransferService.log" 
| rex "\sERROR\s-\s(?<errType>[^\.\r\n:]+)(?<errMsg>.*)"
| eval niceTime = strftime(_time, "%F %T")
| stats list(niceTime) list(errMsg) by errType

This should result in an output like

Unexpected Error        2014-06-20 11:22:23     com.eMeter.PIPe.datatransferblahblahblahb
                        2014-06-21 12:23:34     some.other.errormessage
Invalid date            2014-06-23 22:32:21     Blah Blah Blah error

and so forth. Maybe something like that is what you're looking for?

/K

kristian_kolb
Ultra Champion

btw, if you're NOT doing the stats list(niceTime) part, you can skip the preceding eval niceTime strftime(_time, "%F %T") as well.

EDIT : missing "NOT"
/k

Bergans
Engager

Thanks a lot @kristian.kolb,
I see that I was a bit unclear about what really wanted to achieve. I changed your last part:

| stats list(niceTime) list(errMsg) by errType"

with:

| timechart span=1d count by errType usenull=f

Then I got exactly what I needed.
Now I got a sorted list of the different kind of errors, and a visualized graphical view.

0 Karma

strive
Influencer

Can we consider the following as error messages
Unexpected error com.eMeter.PIPe.datatransferservice.exception.DTSRuntimeException
Invalid date. java.lang.NullPointerException
SDP lookup failed [null]

Means, all error messages start after - and end before :

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...