Splunk Search

show one instance of an error (out of many errors coming repeatedly) from a given time

sajjadkernel
Engager

I am getting many errors while just writing keyword error when searched from a single log file like Retrying connecting ES, AutoReconnect, AttributeError, etc

I want to take out the distinct errors, and every time I had to go and write all the errors in search bar with "AND NOT" keyword just to figure out if there is new error came apart from the listed above, is there a way just to output these distinct errors like select distinct query of SQL?

I tried out different queries from this forums from other threads but none of them seemed to work like transaction, dedup etc.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

See if this gets you started. The rex command attempts to extract the relevant error message text based on your examples. You may need to adjust the regex string to match a larger data set. The results will be a list of unique msg strings and counts for how often each was seen.

host=uat-server source=/var/log/elapp/api/worker.log error | rex "]\s+(?<msg>[^\[\(]+)" | stats count by msg
---
If this reply helps you, Karma would be appreciated.
0 Karma

Lazarix
Communicator

Can you post an example of:

  • The query you are running
  • The results that are returned
  • What you would actually like returned

An example for me is if I do the following:

index="_internal" clientip="*" info

I get data for splunkd_ui_access and splunkd_access logs.

When I search for:

index="_internal" clientip="*" "/services/server/info"

I instead just get info for the splunkd_access logs.

There are many ways to search for the specific data that you're looking for, but we'll need the context in order to help more 🙂

0 Karma

sajjadkernel
Engager

Thanks for the reply, here is the trail of what I am doing

host=uat-server source=/var/log/elapp/api/worker.log error (within time duration last 24 hours)

this gives out with large number of repeated error logs like below

[2018-08-09 06:49:49,653: ERROR/30541] ES Request Error: [code=ed142], [orig_status=404]
[2018-08-09 06:48:46,445: ERROR/13572] ES Request Error: [code=40d08], [orig_status=404]
[2018-08-09 06:04:57,109: ERROR/4706] [2dfb2838-7ac4-4b74-abb9-b63e9cce28d2] raised unexpected: AttributeError("'NoneType' object has no attribute 'get'",)
[2018-08-09 04:00:42,561: ERROR/4874] [a81f570e-61d9-4441-a46d-32dbbba5ceea] raised unexpected: AttributeError("'NoneType' object has no attribute 'get'",)
[2018-08-09 01:52:50,736: ERROR/4863] [1d783dda-688b-451a-82a0-9c97b9e27eb8] raised unexpected: AttributeError("'NoneType' object has no attribute 'get'",)
[2018-08-09 07:28:45,978: ERROR/4847] [20c9e539-745a-42bd-900f-b1233f680c42] raised unexpected: AutoReconnect([Errno 104] Connection reset by peer)

but just to search out if there is any new error I had to repeatedly search by adding the key word with AND NOT every time to just check out if there is any other error that I have missed to check and report, following is my full query when I finally arrive with no more errors to search

host=uat-server source=/var/log/elapp/api/worker.log error AND NOT "ES Request Error" AND NOT AttributeError AND NOT "Historical/Future Index found" AND NOT AutoReconnect AND NOT "Retrying connecting ES"

so if there was a spunk query that can give me all the distinct error logs (cause they are all the same just the time stamp is different) and I don't have to go on and on to build up my search query with AND NOT because every error has its own further course of action so its imp to get all the errors.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...