We're looking at an initial scale-out of Splunk, beyond the traditional, initial single Splunk Enterprise search head + indexer in one. We've essentially been given two servers to move our current server to. I can think of a few permutations, but I'm wondering what's a recommended configuration:
server 1: indexer, server 2: search head
server 1: indexer + search head, server 2: search head
server 1: indexer + search head, server 2: indexer + search head
server 1: indexer, server 2: indexer + search head
All our forwarders are under provisioning automation, so changing the configuration to load balance to two servers shouldn't be a problem, and either way, we have to push out a configuration change for a new indexer. I'm not sure how load balancing to the search heads works - but we have the ability to setup a LB pool on the F5, if that makes sense.
Our indexing license is 100GB/day, and the two server nodes are VMs in a larger vSphere/vBlock deployment. 4 CPUs x 8GB RAM, each. Each has ample disk space to be an indexer. We run a fair number of scheduled searches and when we're investigating an issue / outage, we run a fair number of real-time searches, which tend to start bogging down the current single server we have. I've noticed high load on that server when we run a one-shot on the forwarding servers that pushes a lot of data at one time to the indexer as well.
So, that's all to say, it's not clear to us if we need more indexing capacity, more search head capacity, or just more of both. I'm kind of leaning to the latter. For this reason, we're thinking option #3 (above) might be the place to start, until we have a clearer indicating one way or the other.
I can't find clear docs indicating if #3 is a valid configuration or what to put in place to make it work. Ultimately, we want to be able to still search across the entire index and share the search load across both servers.
Thanks!
p.s. I thought I posted this a couple days ago, but it's not showing up in my profile or in searching answers.splunk.com. Sorry for the repost if it did in fact post mysteriously.
... View more