Deployment Architecture

Why is only one of my indexers showing a high CPU load percentage?

MousumiChowdhur
Contributor

I have six indexers in cluster and 2 search heads. Only one of my indexers is showing >90% CPU load in some other server monitoring tool even when all my inputs are disabled. Also when I run the top command to check the CPU usage of that particular indexer when all my inputs are disabled it shows only 3% to us. What's the best way to check the CPU load and where am I going wrong?

0 Karma

ddrillic
Ultra Champion

We have a similar situation. Looking at the distribution of the daily license across the six indexers helped us a lot. It's on the license master server at licensing.

I assume you would expect an even distribution of the license across the six servers. It's interesting to see what it would show you...

0 Karma

skalliger
Motivator

Here is what I would do to identify the issue.

  • Go to your master and take a look at the DMC, specifically Indexing -> Performance -> Indexing Performance: Instance (select affected indexer)
  • Compare other values from the DMC's Indexing pages with other indexers.
  • Are the indexes configured correct? If you're talking about 6 indexers, I would check the replication settings and also the replication of all indexes. Are they all being replicated?
  • Are you using VMs? If not, check disk health and IOPS, e.g. with Bonnie++
  • In the DMC, you can also take a look at Resource Usage: Instance or Resource Usage: Machine to get some further information.
  • Look at htop/iftop when load is high to get some information which process causes the load which or server is being talked to.

Skalli

0 Karma

renjith_nair
Legend

On the server, if you see a low CPU usage then check your monitoring service to make sure that it's reporting correctly.

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma

MousumiChowdhur
Contributor

The CPU load is is showing up a high percentage >90% on that indexer. How should I check the CPU load and make it less than 90% like my other five indexers? How to find out what is going wrong with that only indexer?

0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...