Which 'should' perform better? Multiple smaller searches time frames or single large search time frames

[Lucas_K]

I have a search performance issue that i'm running into with larger time frame searches in which the time taken is not a linear function of the number of events.

I have a dashboard that populates using a user defined time picker.

Time taken to load depending on time range selected.
1 day = 4-5 seconds.
7 days = 20 seconds.
30 days = 288 seconds.

Anything that takes longer than about 30 seconds is pretty much a candidate for summary indexing so i'll ditch the 30 day scenario for the moment.

I've checked to see which buckets these events are searched from and have determined that the performance difference between hot/warm and cold is not that significant. By this I mean that I can look for events for a single day 30 days ago and its about the same time

So my experiments into improving performance by splitting the searches so they use more cpu's (foo and bar are changed each time as to try and not hit cached events).

A multi append search
ie.

index=blah sourcetype=blah "foo" "bar" earliest=-2d@d latest=-1d@d | eval series="SearchNumberOne"  
| append [ search index=blah sourcetype=blah "foo" "bar" earliest=-3d@d latest=-2d@d | eval series="SearchNumberTwo" ] 
| append [ search index=blah sourcetype=blah "foo" "bar" earliest=-4d@d latest=-3d@d | eval series="SearchNumberThree" ] 
| append [ search index=blah sourcetype=blah "foo" "bar" earliest=-5d@d latest=-4d@d | eval series="SearchNumberFour" ] 
| append [ search index=blah sourcetype=blah "foo" "bar" earliest=-6d@d latest=-5d@d | eval series="SearchNumberFive" ] 
| append [ search index=blah sourcetype=blah "foo" "bar" earliest=-7d@d latest=-6d@d | eval series="SearchNumberSix" ] 
| append [ searchindex=blah sourcetype=blah "foo" "bar" earliest=-8d@d latest=-7d@d | eval series="SearchNumberSeven" ] 

As expected this runs poorly due to the base search waiting on subsearches to return. Time : 41 seconds. Now I don't know if each of the subseaches run sequentially or in parallel in comparison to each other. I do know the base search will wait on the subsearches to return their events.

I've also tried separating the search so that the dashboard spawns multiple searches in different time frames.

Dashboard 1. 1 single 7 day search.
Dashboard 2. 7 single 1 day (different time frame) searches.

The result.
Dashboard 1 takes ~ 19 seconds to load.
Dashboard 2 takes 6 seconds in total to load!(each individual search differs by 1-2 seconds).

This surprised me as I would have thought that the indexers should be working more efficiently doing a larger time scale than having separate jobs that look at events close to each other.

Based off the time taken for dashboard #2 it would suggest that the indexers CAN retrieve all events within the -7d@d time span within 6 seconds its just the method by which its split for that single search is not as resource efficient (be it cpu or disk issues). The down side is the expense of blasting both the search head and index with multiple searches at the same time.

So my question ... 100 duck sized horses (smaller time frame searches) or 1 horse sized duck (1 massive time frame search)?

[HiroshiSatoh]

I think "Search job inspector" and give us a hint.Or, to present the results of the "Search job inspector", it might be better.By the way, Is "larger time frame searches" in my results.

[HiroshiSatoh]

For AJAX,please try to read, if you do not know.

http://dev.splunk.com/view/SP-CAAADSY

[HiroshiSatoh]

Interest has been in the performance of the Splunk during the parallel processing. Do not monitor resources while loading the dashboard "(SoS) Splunk on Splunk"?

[Lucas_K]

The only difference in the job inspector is time taken for dispatch.stream.remote and that is because the time frames is longer.