Splunk Search

How to prevent "max concurrent searches reached" messages on the distributed management console?

a212830
Champion

I've configured a dev Splunk 6.4 env, and noticed that my Distributed Management Console is getting "max concurrent searches reached" messages. Sooooo, since the DMC isn't part of the Search Head Cluster and can't benefit from the improved scheduling, and since the DMC is probably going to add more and more searches over time as it develops... I'm looking for idea on how to prevent this situation, since the DMC is supposed to be runnable from a "smallish" VM (6 vcpu's).

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

DMC fires searches only when you open a page or a dashboard. We try to limit the total number of searches on each page/dashboard.

Reaching max concurrent searches wouldn't prevent DMC from working, the searches are just in the waiting queue and will run as soon as other searches finished.

0 Karma

dfqobvbkmnpi
Explorer

what about the DMC alerts?

0 Karma

a212830
Champion

True, but if it's a trouble-shooting tool, then I need the most recent information, and I might not have it. Seems to be an area that was overlooked, in all honesty... more and more searches are going to be added to the DMC, which is going to require it to be a bigger server.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I agree with @ykou. I also run my DMC on a tiny VM and ALWAYS run out of concurrent searches when I load the DMC's default dashboard. I trust that they'll just queue up and I don't mind waiting a moment so I ignore it.

It's merely informational to let you know that the other searches are going to queue since you hit the max. I only really see it on that first dashboard since so much happens on there.

If its truly an issue, you could increase the DMC cpu's or modify the per cpu limits.conf setting. I'm not sure its worth it though.

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...