Splunk Search

Search query taking ages to finish in Splunk 7.0.1

AKG1_old1
Builder

Hello,

I have upgraded Splunk Enterprise to 7.0.1. One of the search query is taking ages to finish it. Same query finished quickly in Splunk 6.x.

Splunk 6.6.1 = 5 secs
Splunk 7.0.1 = 26 mins (still running)

Does anyone have encounter such situation or have idea for this behaviour in Splunk 7.

alt text

sarfarajsayyad
New Member

In my case, I have added "| noop search_optimization=false" at the last line which makes my query fast. You can give a try.

0 Karma

amaind1990
New Member

@sarfarajsayyad @agoyal I am also getting same issue. Were you able to resolve the issue? If yes, what is the solution? Is this issue is specific to new version 7.1.0?

0 Karma

sarfarajsayyad
New Member

In my case I have added "| noop search_optimization=false" at the end of the line to resolve the issue. It worked for me, you can give a try.

0 Karma

niketn
Legend

@agoyal, you second search should be part of your base search

<yourBaseSearch> "*concurrent mode failure*"
| stats count as FAIL

Have you tried it this way?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

sarfarajsayyad
New Member

Same behavior I see on my setup. My query was running fine on 6.5.2 (within 2 mins) and now it is taking more time (90 mins) on 7.0.2. Is there any change happened on 7.0.x? I don't see any error msg on splunkd log too. There is no clue.

0 Karma

niketn
Legend

@sarfarajsayyad , would it be possible for you to share the query?

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

sarfarajsayyad
New Member

That query is having some customer specific info. Cant share here.
My point is, the same query is running fine on 6.5.2 and not on 7.0.2. Believe me, I have installed 7.0.2 Splunk enterprise on a new machine without data. Still, it's taking more time. Looks like its nothing related to data. Something is changed in 7.0.x.

JFYI - In my query, I have 20+ joined on various indexes/lookup.

0 Karma

niketn
Legend

@sarfarajsayyad if you have valid Splunk Entitlement, reach out to Splunk Support.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

sarfarajsayyad
New Member

If I use "|noop search_optimization=false" at the end of my query its giving me result very fast. After cross-checking I got the same result.

Is there any impact of "|noop search_optimization=false" ?

0 Karma

sarfarajsayyad
New Member

I see lot of below messages

02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='inputlookup' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='eval' doesnt have raw field
02-22-2018 04:06:41.512 INFO SearchPipeline - Command='search' doesnt have raw field
02-22-2018 04:06:41.512 INFO SortOperator - maxmem = 209715200

0 Karma

mayurr98
Super Champion

In the first search its written before 17/1/2018 from how long do you have data in your system?

0 Karma

AKG1_old1
Builder

file is created on 17/01/2018 06:27:12. So its having data for few hours.

same file is taking by both Splunk version. I am having feeling that as time is specified as "All Time" , Splunk 7 is not specifically looking for specified file. It's trying to find out data from the begnning of Splunk Time (1st Jan 1970).

Search is still running.

elliotproebstel
Champion

That's correct. There are a small handful of fields that are extracted at index-time, and unless you've done some intentional work to change that on your system, then the field service_name will not be one of them. So now Splunk is searching back through all indexed data looking for any possible matches. Your best bets to make this search run more efficiently are to narrow the time window and specify the index(es) in which you'd like to search. Here's a good guide to provide more of an overview:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Search/Writebettersearches#Tips_for_tuning_yo...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...