Indexer is going down when running large number of searches

[jet1276]

Hi,

I have one indexer and 3 search heads in cluster mode. And I have developed too many dashboards with graphic representation.

Now when ever i open 2 or more dashboards from my search head, which will be having total 30 or 40 searches, my Indexer is gong down for some time and again its coming back up.

Is there any connection restriction from Search Head to Indexer? Or Could this be some other issue?

[gcusello]

Hi jet1276,
if you run a search, this takes a CPU and some memory and releases them only when finished, so having many searches at the same time probably you finished your resources and the other searches are queued.
Are any searches in real time? if yes these searches don't release hardware resources!

You should make a capacity planning of your infrastructure because probably your infrastructure isn't sufficient to support you load.

You could have an idea of your activity using Distributed Management Console or installing Activity Search App.

Bye.
Giuseppe

[jet1276]

Hi @cusello,

I am having system with large resources and there are sufficient resources available for the new queries to run.

Also I confirmed that by opening dashboards and putting Indexer in Down state while monitoring CPU and Memory resources.

CPU utilization was below 50% and Memory still 2 to 3 GB was available during that period.

[gcusello]

Hi jet1276,
What do you mean with large resources?
if you use at the same time two dashboard with 30 searches each one you have at the same time 60 searches! have you 60 CPUs on you Indexer?
I had a similar problem because one of my customers had around 10 users and each one used one or two dashboards with 12 real time searches, globally I had 87 contemporary real time searches!
I replaced them with scheduled reports and I grouped searches in each dashboard using Poste Process Searches, reducing load on my indexers.

Use DMC to monitor your situation when you have problems.
Anyway I suggest to open a Case to Splunk Support but they probably will say to you the same thing!

Bye.
Giuseppe

[jet1276]

Hi @cusello,

I have 48 CPU server and its utilization max goes to 60%. And I have changed settings of "max_searches_per_cpu" in limits.conf to 2. So at least 96 realtime searches can run at a time.

So thats why I think I have enough resources.

And I do not have DMC at the moment but I will integrate it in some time and check the utilization again.

Please let me know if I need to do anything else.

Thanks.

[kunalmao]

you are wrong to assume that it will support 96 real time searches by just having that seeting

[gcusello]

Open a case to Splunk Support: they will support you.
Bye.
Giuseppe

[jet1276]

Thanks so much @cusello for your help.

[aphilip]

Hello jet1276,

Has this issue been resolved? I'm having similar issue with 1 search and 1 indexer where the indexer goes down(8000 and 8089, but Splunk status show up) when large searches are performed and coming back up after a while. Both search head and indexer servrs are running 7.1.

Thanks.