Deployment Architecture

Why is only one of my indexers showing a high CPU load percentage?

MousumiChowdhur
Contributor

I have six indexers in cluster and 2 search heads. Only one of my indexers is showing >90% CPU load in some other server monitoring tool even when all my inputs are disabled. Also when I run the top command to check the CPU usage of that particular indexer when all my inputs are disabled it shows only 3% to us. What's the best way to check the CPU load and where am I going wrong?

0 Karma

ddrillic
Ultra Champion

We have a similar situation. Looking at the distribution of the daily license across the six indexers helped us a lot. It's on the license master server at licensing.

I assume you would expect an even distribution of the license across the six servers. It's interesting to see what it would show you...

0 Karma

skalliger
Motivator

Here is what I would do to identify the issue.

  • Go to your master and take a look at the DMC, specifically Indexing -> Performance -> Indexing Performance: Instance (select affected indexer)
  • Compare other values from the DMC's Indexing pages with other indexers.
  • Are the indexes configured correct? If you're talking about 6 indexers, I would check the replication settings and also the replication of all indexes. Are they all being replicated?
  • Are you using VMs? If not, check disk health and IOPS, e.g. with Bonnie++
  • In the DMC, you can also take a look at Resource Usage: Instance or Resource Usage: Machine to get some further information.
  • Look at htop/iftop when load is high to get some information which process causes the load which or server is being talked to.

Skalli

0 Karma

renjith_nair
Legend

On the server, if you see a low CPU usage then check your monitoring service to make sure that it's reporting correctly.

Happy Splunking!
0 Karma

MousumiChowdhur
Contributor

The CPU load is is showing up a high percentage >90% on that indexer. How should I check the CPU load and make it less than 90% like my other five indexers? How to find out what is going wrong with that only indexer?

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...