Monitoring Splunk

Splunk Instance Performance : at Large Scale

clyde772
Communicator

Hey guys,

This questions is to those who had a little in-depth splunk engineering experiences.

I am trying to design a optimal architecture for our splunk installation. Here are my objective.

Environment :
On a well designed H/W, 16 Cores (Linux OS), CentOS, total approximately 4000 IOPS storage system

  1. Will a single Splunk instance be able to consume (well utilize) majority of CPU and 4000 IOPS of storage? in other words, In order to consume as much IOPS possible with a single SPLUNKD instance? Of course to maximize index volume + maximize search performance.
  2. Do you prefer multiple splunk instances per server vs. single instance per server? Pros and Cons. I hear mixed opinions as well as mixed performance test results.

I appreciate any real life experiences anyone of you had running Splunk in a big scale deployments.

Cheers~!

Tags (1)
0 Karma

bmacias84
Champion

I going to say this all really depends if you are talking about distributed search set up or an all-in-one.

Questions you have got to ask is how much through put are you estimating? How many dashboards will you have with how many searches? Will you have adhoc searches?

Ever search cost 1 CPU for the duration of the search. Now lets say each dashboard use 6 searches costing 6 CPUs (if your not using post process searches), then you have 3 saved search at any time, then you have 4 real time dashboards at 2 searches a piece running always occupying that's 8 more and not including adhoc searches. If that was happening constantly that's 15 concurrent searches using 15 CPU. That's worst case scenario. Now take into effect indexing more CPUs.

Worst case you also use the max of 200 MB of memory per search that's 3GB just in searches.

Regarding IOP, those will be consumed my searches and indexing. Your index could be affected by how intense your searches are and vice versa.

These are just worst cases. Cheers

Read this doc for more info on Scaleyourdeployment

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

An indexer will use as many IOPS as your hardware can give to it. It will not utilize that many cores, the key is write performance. Your hardware should be capable of well over 50GB/day indexing volume.

A search head will use those cores - provided there are a number of users/searches running in parallel, those usually are more CPU-intensive than indexing.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...