Splunk Search

Splunk maxing out at 11 realtime searches despit having 16 CPUs

JeremyHagan
Communicator

Hi,

I have a single-server instance of Splunk with 16 cores. According to my research the maximum number of realtime searches should be:
max real-time searches = max_rt_search_multiplier x max historical searches
and
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches

So substituting in values from limits.conf I see:
max real-time searches = 1 x (1 x 16 + 6) = 22

But in my scheduler.log I get:
The maximum number of concurrent scheduled searches has been reached (limits: historical=11, realtime=11). historical=0, realtime=12 ready-to-run scheduled searches are pending.

Suspiciously these two numbers add up to 22. What am I missing here? I thought realtime should be 22.

0 Karma
1 Solution

Masa
Splunk Employee
Splunk Employee

I believe it is a simple reason. scheduled searches is 50% of total historical search or realtime search. In your case, hist search max is 22. So, scheduled search is 50% of 22 = 11. Same for real-time search. 11 is max for real-time scheduled searches. It is in limits.conf.spec file.

Old info in wiki.splunk.com still works in general.
http://wiki.splunk.com/Community:TroubleshootingSearchQuotas

View solution in original post

Masa
Splunk Employee
Splunk Employee

I believe it is a simple reason. scheduled searches is 50% of total historical search or realtime search. In your case, hist search max is 22. So, scheduled search is 50% of 22 = 11. Same for real-time search. 11 is max for real-time scheduled searches. It is in limits.conf.spec file.

Old info in wiki.splunk.com still works in general.
http://wiki.splunk.com/Community:TroubleshootingSearchQuotas

JeremyHagan
Communicator

Sound plausible. From the [scheduler] section:
max_searches_perc: defaults to 50.

I ended up changing the rt search multiplier from 1 to 2. I can't find any good data on what value it is safe to set this to. Some people have said 4 and one person set it to 8. Setting it to 2 got me out of trouble in this case.

I wish the documentation was clearer on this.

0 Karma

Masa
Splunk Employee
Splunk Employee

Once you changed the default value, you are easier to reach high resource usage and performance issue. It all depends on how your searches finishes quickly and avoid long running concurrent searches. I had a user standalone and set to 4 and once enabling Report acceleration (a lot), their end users complained performance issue. They changed to 2. Still not enough to avoid performance issue. They had to disable Report acceleration. Again, basically, changing the default value will push system resource work harder. How much hard they can work without bad user experience? That all depends.

0 Karma

inventsekar
Ultra Champion

in my scheduler.log I get:
The maximum number of concurrent scheduled searches has been reached (limits: historical=11, realtime=11). historical=0, realtime=12 ready-to-run scheduled searches are pending.

i think, the (limits: historical=11, realtime=11) was misleading.
it should have said - (limits: historical=11 OR realtime=11). we should not add 11+11=22. its either 11 historical or 11 real time searches.

update -

can you run this for last 24hrs or last 7 days and update us few results...

index=_internal sourcetype=splunkd source=*metrics.log group=search_concurrency "system total" | table active_hist_searches active_realtime_searches

JeremyHagan
Communicator

But why is it 11? According to the doco it should be 22.

0 Karma

inventsekar
Ultra Champion

may we know your limits.conf [search] configuration please.
also is this on search head cluster? ur splunk version also, please

0 Karma

JeremyHagan
Communicator

And it is a single-server instance, so indexer and search head all on one server. Running Splunk 6.2.6

0 Karma

inventsekar
Ultra Champion

can you run this for last 24hrs or last 7 days and update us few results...
index=_internal sourcetype=splunkd source=*metrics.log group=search_concurrency "system total" | table active_hist_searches active_realtime_searches

0 Karma

JeremyHagan
Communicator

Even for the last 4 hours that returns 240,000+ results and they seem to read 0,0.

0 Karma

JeremyHagan
Communicator

If I sort by -active_realtime_searches then I get 13 as the max number of realtime searches in the last 24 hour period and 15 for historical.

I do get a row of 15 historical and 11 reatime

0 Karma

inventsekar
Ultra Champion

15 and 11 on the same line ah?!?
totally 26 searches ah?!?
One more question - how you found out the CPU cores please.. are you sure about the number of CPU cores on this system?!?

0 Karma

JeremyHagan
Communicator

I'm 100% sure about the number of cores. Checked through Task Manager on Windows, plus I administer the VM it is running on and it has 2 x sockets with 8 cores each presented to it.

0 Karma

inventsekar
Ultra Champion

Oh Ok, great..
May I know, 15 and 11 on the same line ah?!?
totally 26 searches ah?!

0 Karma

JeremyHagan
Communicator

Correct. On the same line.

0 Karma

inventsekar
Ultra Champion

Great, though I didn't solve this, I am glad, we troubleshooted this.. maybe, an upvote, to cheer me;)

0 Karma

inventsekar
Ultra Champion

But in my scheduler.log I get:
The maximum number of concurrent scheduled searches has been reached (limits: historical=11, realtime=11). historical=0, realtime=12 ready-to-run scheduled searches are pending.

Means, its saying only 11 and 11 searches together.. but, above one says 26 searches?!?

0 Karma

JeremyHagan
Communicator

From running splunk.exe show config limits I get the following for the [search] stanza. To my knowledge these are all defaults:
[search]
allow_batch_mode=true
allow_inexact_metasearch=false
base_max_searches=6
batch_retry_max_interval=300
batch_retry_min_interval=5
batch_retry_scaling=1.5
batch_search_max_index_values=10000000
batch_wait_after_end=900
cache_ttl=300
chunk_multiplier=5
default_allow_queue=true
default_save_ttl=604800
dispatch_dir_warning_size=2000
dispatch_quota_retry=4
dispatch_quota_sleep_ms=100
enable_history=true
failed_job_ttl=86400
fetch_remote_search_log=disabledSavedSearches
fieldstats_update_freq=0
fieldstats_update_maxperiod=60
load_remote_bundles=false
long_search_threshold=2
max_chunk_queue_size=1000000
max_combiner_memevents=50000
max_count=500000
max_history_length=1000
max_id_length=150
max_macro_depth=100
max_rawsize_perchunk=100000000
max_results_perchunk=2500
max_rt_search_multiplier=1
max_searches_per_cpu=1
max_tolerable_skew=60
max_workers_searchparser=5
min_freq=0.01
min_prefix_len=1
min_results_perchunk=100
multi_threaded_setup=0
preview_duty_cycle=0.25
queued_job_check_freq=1
realtime_buffer=10000
reduce_duty_cycle=0.25
reduce_freq=10
remote_timeline=true
remote_timeline_connection_timeout=5
remote_timeline_fetchall=1
remote_timeline_min_peers=1
remote_timeline_receive_timeout=10
remote_timeline_send_timeout=10
remote_timeline_touchperiod=300
remote_ttl=600
replication_file_ttl=600
replication_period_sec=60
result_queue_max_size=100000000
results_queue_min_size=10
rr_max_sleep_ms=1000
rr_min_sleep_ms=10
rr_sleep_factor=2
search_process_mode=auto
stack_size=4194304
status_buckets=0
status_cache_size=10000
summary_mode=all
sync_bundle_replication=auto
target_time_perchunk=2000
track_indextime_range=true
truncate_report=false
ttl=600
write_multifile_results_out=true

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...