Good setting for max_searches_per_cpu?

[echalex]

Hi,

I'm getting this warning every hour, on top of the hour, when apparently quite a few scheduled searches are triggered.

The system is approaching the maximum number of historical searches that can be run concurrently. current=19 maximum=20

Now, I understand that this does not necessarily cause a problem, since the searches will be slightly postponed. Correct? However, the CPU usage on all cores for both search heads is very close to 0%, constantly. So it seems that increasing max_searches_per_cpu slightly could be appropriate, but I can't find documentation explaining what good values are.

The CPU model is Intel(R) Xeon(R) CPU E7- 2870 @ 2.40GHz.
There are 4 cores per search head and 8GB RAM.

Any opinions?

[dbylertbg]

As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.

[yannK · Accepted solution]

The default setting is max_searches_per_cpu=4 and is already optimized. Increasing it may cause your searches to slow down or worse.

The best solution to avoid to reach the max, is to change the schedule of your searches. Spread them other different intervals.

Example, for an hourly search over the last hour :
cron = 0 * * * * earliest=-1h latest=now

can be replaced by a search that starts 5min after the hour :
cron=5 * * * * earliest=-1h@h latest=@h

[dbylertbg]

As of Splunk 5.x it appears that max_searches_per_cpu=1 is the default due to the addition of good job queuing for GUI searches. See This answer for more details.

[echalex]

On the other hand, I do feel that the cpus are heavily under-utilised. Whenever I get this warning, the cpus are nearly idle.

So, unless the problem is between the search head and the indexers, I don't really see why a slight increase would be a problem... Doubling the cores would remove this warning, but would also result in more idle cpus.

[yannK]

to have an idea of the scheduling searches, take a look at the SOS app
http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk

the searches dashboard shows the scheduling and skipped searches.

[echalex]

Ok, thanks!

Rescheduling sounds doable. The only hitch is that users can schedule searches, so they may not be aware of this issue.