Splunk Search

Why are small searches taking incredibly long?!

paimonsoror
Builder

I have been hunting down users in my environment running real-time searches as I thought that they were the root cause, but I have also seen from time to time searches like the ones below lingering around hte Jobs queue. The impact of course is that we start to hit our concurrent search limit

alt text

Looking at the logs for these searches, majority of them seem to be stuck in the following loop:

10-11-2016 21:23:18.306 INFO  UnifiedSearch - Processed search targeting arguments
10-11-2016 21:23:18.306 INFO  DispatchThread - Generating results preview took 1 ms
10-11-2016 21:23:23.306 INFO  SearchParser - PARSING: timechart  count
10-11-2016 21:23:23.306 INFO  bucket - Setting info._summary_maxtimespan = 5s
10-11-2016 21:23:23.306 INFO  SearchParser - PARSING: stats count AS query| search query = 10
10-11-2016 21:23:23.306 INFO  StatsProcessor - No group-by fields specified, incompatible for high cardinality improvements

The search that the log above came from is the following:

source="SOME_SOURCE_LOG_FILE" | regex _raw=".Error.|.error.|.Exception.|.exception."|timechart count
0 Karma

woodcock
Esteemed Legend

The problem is the regex command. Try this search instead:

 source="SOME_SOURCE_LOG_FILE" (*error* OR *exception*) | timechart count
0 Karma

paimonsoror
Builder

Thanks for this. Would you mind elaborating? Why would the regex cause this?

0 Karma

lycollicott
Motivator

If you run that search for the last 60 minutes, how many events does it return? How many different hosts do they come from?

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...