Horizontal or Vertical scaling for best search head performance

[richnavis]

Hello, I've already read the Splunk planning for a large scale deployment documents. However, I didn't get a sense about what would be better for scaling searchheads... adding more servers or adding processors to existing servers. We have a lot of utility hardware and are debating whether we'd get better performance by adding 4 searchheads with 2socket/4core procs, or adding adding 2 searchheads with 4 socket 4 procs, given equal memory/processor speed and ignoring the differences in OS and Hardware Management, does anyone see a PERFORMANCE BASED reason to choose one or the other?

[Damien_Dallimor]

As dwaddle states, there are also HA considerations.
But from a performance point of view , what are the expectations with respect to :

1) max concurrent users

2) max concurrent searches(inline and scheduled)

[richnavis]

Sometimes we have 20+ users doing needle in haystack type searches. From a concurrent searches perspective, we make extensive use of Views and scheduled searches.. so sometimes close to 40/50 concurrent searches..

[dwaddle]

Well, this isn't strictly a performance concern, but with more horizontal search heads you either need search head pooling (which you may wind up with using 2 search heads anyway, depending on how highly available you wish do configure) -- and this brings with it the need for highly-available, high-performance NFS. All things being otherwise equal, I think my preference would be for two larger search heads.

[richnavis]

Thanks for the reply... Yes, I am aware of the HA considerations, and the complications of Search head pooling. In order to simplify the answer, I tried to have the focus be on "Performance based reasons" only..