Splunk Search

Why is my scheduled search so much quicker than my adhoc search?

a212830
Champion

Hi,

I have a number of scheduled searches which run significantly faster than the same search run from the search-bar. I have no idea why this would happen - are there some settings that might cause this? The scheduled search is about 30-40 seconds (runs every 30 minutes throughout the day). The same adhoc search runs for minutes.

Search:

index=raytheon_proxy dstBytes>0
|eval totbytes=bytes/1024/1024/1024
|chart sum(totbytes)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

First, try running the search with these settings -

1) The same time boundaries (probably earliest=@d latest=@m or earliest=-30m@m latest=@m)
2) set to fast, NOT set verbose.


Second, take a look at the job inspector for each job and see what is different.


Third, do this and see if anything changes. (It shouldn't, since the optimization should take care of that)

 index=raytheon_proxy dstBytes>0
 | fields bytes
 | eval totbytes=bytes/1024/1024/1024
 | chart sum(totbytes)

Fourth, see if you can get the same information from tstats, which would save massive amounts of search time. That would look something like this...

 | tstats sum(bytes) as bytes where index=raytheon_proxy dstBytes>0
 | eval totbytes=round(bytes/1024/1024/1024,2)

Fifth, not sure why you are testing dstBytes and summing bytes, but hopefully you do.

View solution in original post

DalJeanis
SplunkTrust
SplunkTrust

First, try running the search with these settings -

1) The same time boundaries (probably earliest=@d latest=@m or earliest=-30m@m latest=@m)
2) set to fast, NOT set verbose.


Second, take a look at the job inspector for each job and see what is different.


Third, do this and see if anything changes. (It shouldn't, since the optimization should take care of that)

 index=raytheon_proxy dstBytes>0
 | fields bytes
 | eval totbytes=bytes/1024/1024/1024
 | chart sum(totbytes)

Fourth, see if you can get the same information from tstats, which would save massive amounts of search time. That would look something like this...

 | tstats sum(bytes) as bytes where index=raytheon_proxy dstBytes>0
 | eval totbytes=round(bytes/1024/1024/1024,2)

Fifth, not sure why you are testing dstBytes and summing bytes, but hopefully you do.

a212830
Champion

Thanks. We are running 6.5, so the optimization isn't available yet. The third option was huge, should have realized it. The fourth, unfortunately is not available.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

@DalJeanis with the clutch search tips as always!

I can't keep my mouth shut so here's another detail:

index=raytheon_proxy dstBytes>0
| stats sum(dstBytes) AS bytes
| eval TBs = bytes / pow( 1024 , 3 )

The point being that the search will be faster if the eval has less fields to compute on (such as after a stats transformation) and I find the pow easier to read then debugging vs having to count the # of times I wrote 1024.

BUT, this is tangential relative to why the search speeds are so different. Like was mentioned, check Job Inspector!

DalJeanis
SplunkTrust
SplunkTrust

Nice point. Dividing three separate times by 1024 is inefficient, isn't it? If there were more than one event at that point - for instance, if we were calculating MB on the individual events for some reason - then it could save significant CPU cycles by multiplying pow(1024,3) out into a constant...

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Oh, actually I was more emphasizing the use of stats and then eval.

Here's an example: we have one million events and we use stats to get it down to ten results. If we do eval then stats then we're calculating eval on a million events. If we flip it, we'll be doing eval on ten events. Furthermore, stats is a reducing command, so the payload sent back from the indexers should be summarized to just what's needed. eval is a streaming command and will obligate the indexers to send back full payload fidelity of the events and then make the search head use more memory to retain those events for the eval, only after which the stats happens (on the search heads).

Hopefully that clears up confusion I introduced.

DalJeanis
SplunkTrust
SplunkTrust

@sloshburch - Got it. One slight adjustment to wording ..., the eval before a stats is streamable, so it is a million calculations done at the indexers, not at the search head. Nonetheless, with those ratios, you want it done at the search head. If the average ratio in your data were, say, five to one, and you had 50 indexers, then you'd want it done at the indexers.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Great point! Thanks for correcting me. Def not a typo but a full on mistake by me. Great correction and thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...