Deployment Architecture

Does Splunk use more than 4 cores?

dhaffner
Path Finder

I’m hoping someone can help answer this.
We have seen and heard some bits and pieces about Splunk really only using up to 4 cores on a Linux machine. Is this true? What is the real limit?
Basically, is it worth getting an indexer with 16 cores, or even 24? We are getting ready to order 3 new DL580’s for our environment and have been given the option of 24x128 machines. Is that just crazy overkill for an indexer? Is there any documentation that directly addresses this? I haven’t found much of anything other than a couple things here on answers, which don’t say for sure.

Thanks!

Tags (1)
0 Karma
1 Solution

tedder
Communicator

We have Splunk split across a handful of 16-core servers. Searches are single-threaded, so the determining factor is the number of concurrent users/searches.

Our experience is that we are much more IO bound than CPU-bound.

View solution in original post

0 Karma

araitz
Splunk Employee
Splunk Employee

That is likely crazy overhead. Given how you guys use Splunk, you would be better off going for 3 indexers with 8 cores rather than one indexer with 24 cores.

You can install multiple instances of Splunk on a machine, and to some extent will experience better individual search performance. However, you will pay for it in terms of additional management complexity, increased contention, and adding a huge single point of failure.

tedder
Communicator

We have Splunk split across a handful of 16-core servers. Searches are single-threaded, so the determining factor is the number of concurrent users/searches.

Our experience is that we are much more IO bound than CPU-bound.

0 Karma

araitz
Splunk Employee
Splunk Employee

It is easy to confuse cores with processes with threads, but they aren't equivalent. Each search is a separate multi-threaded process. There are certain parts of a search process that are not implemented to use threads, while other parts may leverage multiple threads. That said, it is a safe rule of thumb that one search will use around one core for sizing purposes.

dhaffner
Path Finder

watching a couple indexers each 16x64, we have 6 or 7 splunkd PIDs going and loads around 8. A bit of SWAP is being used, too. That is pretty much normal for us. So I'd assume it would be best to stay with the 16x64 and 100 GB/day. Thoughts?

0 Karma

dhaffner
Path Finder

searches each use a core, right? and each one is sent out to the distributed indexers, right? So how many cores does just indexing use? Is it the indexing that is single threaded?

0 Karma

tedder
Communicator

we should chat offline- they are bound to one core when we watch 'top'.

0 Karma

araitz
Splunk Employee
Splunk Employee

Searches are not single threaded.

dhaffner
Path Finder

Also, if we put more than one instance of Splunk on a 16x64 machine, will they use different cores and be more effective? Or will it just bottleneck at the drive I/O and network?
Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...