Knowledge Management

Tuning max searches on a summary indexing instance - how?

the_wolverine
Champion

I have an instance that I've set up to only run summary searches. Essentially, its a search head but no users connect directly to it and it only runs summary indexing searches.

I see a lot of the following errors in my splunkd.log:

WARN SavedSplunker - Maximum number (2) of concurrent scheduled searches reached. 16 ready-to-run scheduled searches pending.

Can I tune some parameters in limits.conf to better the performance? Right now, its telling me I'm maxing out at 2 concurrent searches and it should be able to handle more considering no users are connecting directly to it.

0 Karma
1 Solution

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

View solution in original post

the_wolverine
Champion

In general, we don't advise that you edit limits.conf unless you really know what you're doing.

In this situation, you should be able to modify the following settings in $SPLUNK_HOME/etc/system/local/limits.conf (listed below are the defaults in version 4.1):

[search]
max_searches_per_cpu = 4

[scheduler]
max_searches_perc = 25

Let's say your instance has 2 cpus. The number of (concurrent) searches per cpu, based on the default settings, will be 8 searches. For scheduled searches the default is 25% of that number so your max concurrent SCHEDULED searches (which applies to summary indexing searches) will be 2 concurrent searches.

If you're only running summary searches on this machine, you could raise the max_searches_perc up to 100 meaning that up to 8 scheduled searches can run concurrently.

If this system is not utilized by anything else, you could potentially raise the max_searches_per_cpu setting as well.

After modifying either of these settings make sure to monitor your system for a period of time to ensure it is not being overtaxed at any point in time.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...