Splunk Search

How to prevent "max concurrent searches reached" messages on the distributed management console?

a212830
Champion

I've configured a dev Splunk 6.4 env, and noticed that my Distributed Management Console is getting "max concurrent searches reached" messages. Sooooo, since the DMC isn't part of the Search Head Cluster and can't benefit from the improved scheduling, and since the DMC is probably going to add more and more searches over time as it develops... I'm looking for idea on how to prevent this situation, since the DMC is supposed to be runnable from a "smallish" VM (6 vcpu's).

0 Karma

ykou_splunk
Splunk Employee
Splunk Employee

DMC fires searches only when you open a page or a dashboard. We try to limit the total number of searches on each page/dashboard.

Reaching max concurrent searches wouldn't prevent DMC from working, the searches are just in the waiting queue and will run as soon as other searches finished.

0 Karma

dfqobvbkmnpi
Explorer

what about the DMC alerts?

0 Karma

a212830
Champion

True, but if it's a trouble-shooting tool, then I need the most recent information, and I might not have it. Seems to be an area that was overlooked, in all honesty... more and more searches are going to be added to the DMC, which is going to require it to be a bigger server.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I agree with @ykou. I also run my DMC on a tiny VM and ALWAYS run out of concurrent searches when I load the DMC's default dashboard. I trust that they'll just queue up and I don't mind waiting a moment so I ignore it.

It's merely informational to let you know that the other searches are going to queue since you hit the max. I only really see it on that first dashboard since so much happens on there.

If its truly an issue, you could increase the DMC cpu's or modify the per cpu limits.conf setting. I'm not sure its worth it though.

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...