Splunk Search

Why do I receive error "system wide historical concurrent searches quota has been reached"?

ankithreddy777
Contributor

we have 10 indexers with 16 CPU cores each. Our replication is 4
base_searches=6 and max_searches_per_cpu =1.

I am getting an error stating
"system wide concurrent searches has be reached, maximum=38 current=38".

I assume that if indexer has 16 cores, system wide limit should = base_searches+max_searches per_cpu*number of cpu=6+1*16 =22

But why I am getting the error with 38 limit?

0 Karma

sowings
Splunk Employee
Splunk Employee

This limit is actually enforced by the search head, not the indexers. The limit of 38 would be reflected as 6 + the number of cores in your search head, which I'm guessing is 32.

0 Karma

ankithreddy777
Contributor

My search head has 16 cores with max_searches_per_cpu setting =1

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Are you using Search Head Clustering?

0 Karma

ankithreddy777
Contributor

we are not using search head clustering

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Can you run following search see what is the numberOfCores identified by Splunk?

| rest /services/server/info splunk_server=local | table host numberOfCores
0 Karma

ankithreddy777
Contributor

Its giving number of cores on server(which is SH)= 16

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...