Getting Data In

Why am I getting a HTTP 503 error when using threads to call oneShotSearch and maximum number of concurrent historical searches?

Aweave15
New Member

Hi All,

I am using the Java splunk api service to make oneShotSearch calls for service data.
HTTP 503 response: Search not executed: The maximum number of concurrent historical searches on this instance has been reached.", concurrency_category="historical", concurrency_context="instance-wide", current_concurrency=34, concurrency_limit=34

What I am doing is creating multiple threads which perform the Service#oneshotSearch in given time intervals. It runs normally, but eventually, I get the HTTP 503 above error. I am not that familiar with Splunk (mainly focusing on the Java code) so could someone please elaborate on what this exactly means?

I am confused what exactly instance is referring to, is this referring to a specific host included in the queries, or is this referring to my own user searches once I am logged in? I do not have 34 threads, so are they getting backed up and queued through the api?

If anyone could please provide me with some details on this error to help troubleshoot, I would greatly appreciate it. Thank You!

0 Karma

493669
Super Champion

Open the job inspector (top right menu) and check how many searches are running.
You may have scheduled searches running
there are 2 limits that may kick in :
- quota of number of concurrent searches per user (see the roles, and authorize.conf)
- quota of the system maximum number of concurrent searches (see number of cores and limits.conf)

And those limits are different for historical searches and real time searches.
see http://wiki.splunk.com/Community:TroubleshootingSearchQuotas
have a look at https://answers.splunk.com/answers/607068/the-maximum-number-of-concurrent-historical-search.html

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...