We are currently wrestling with issues with search head pooling ourselves. We have 3 search heads and had originally planned to put them in a GFS2 cluster using a dedicated SAN LUN shared between all 3 (>5000 IOPS). After finding out that Splunk doesn't support direct storage (which they really should, and we've asked that it be put on the roadmap), we cut the connection for two of the servers, and now have a single server sharing the mounted lun via NFS. With 128 NFS instances and client connections maxed out on other search heads, it still isn't performing well. Support has been helping.
... View more