Getting Data In

Will upgrading our Splunk indexer server with high performance hardware and dividing the indexer with multiple servers solve our high CPU performance issue?

rameshlpatel
Communicator

Hi Team,

Currently we are facing a high CPU utilization issue on our indexer because of Splunk's high usage. To resolve this we are planning to upgrade our server in two ways.

1) Upgrade same server with high performance hardware.
2) Divide indexer with multiple servers.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

You don't have a very complex setup as bmacias84 mentioned... and yet you mention a 20% increase in volume. Was that 150GB per day and it's now gone beyond to 180? Anything less than that might be helped by adding an indexer but it will just be diluting the problem not solving it.

You mention high performance hardware (mem? CPU? etc...)

Unnaturally High CPU usage can = something like bad disk. Indexer IO is quite intensive... so if it's hammering around trying to manage buckets, you're going to see CPU go through the roof. And if there are bad spots and good spots... it won't be consistent. This is a great reason to request support as there are many tools they can use to help you narrow down what might be the problem.
Check out this page here
http://docs.splunk.com/Documentation/Splunk/6.2.4/Troubleshooting/Collectpstacks
Create a diag of each component.
$SPLUNK_HOME/bin/diag
and open a support ticket. Upload the diag and results of pstack (or if on windows follow info on collecting stack bits)

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!

bmacias84
Champion

What does your current topology look like?
How many transforms (index time extractions) be performed?

rameshlpatel
Communicator

I have 1 SH & 1 indexer with 58 forwarders. From Monday size of data coming to indexer has increased upto 20 percent due to some new changes.

0 Karma

bmacias84
Champion

Seem like a relatively small deployment. Whats your indexing volume? What Splunk apps are you using? Was your indexer ever a search head (All-in-one deployment)? Have you configured the distributed management console, it has a lot of good diagnostic information. I would look at CPU Usage per Splunk Processor. More indexers wont hurt, but adding hardware before you know what your bottle neck is won't necessarily fix your issue.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...