How much I/O is used for indexing and searching in a clustered environment? What resources are there for guidance?

[sunrise]

Hi Splunkers,

I'm considering about splunk clustering in VM env, 1 Search Head, 3 Search Peers, 1 Cluster Master.
And Replication Factor = 3, Search Factor =2.
It's VM environment, so I'm afraid that splunk disk I/O usages will take an effect on other VM instances.

For example, in the case of index time,
if we get 128 kb/s input data to a splunk clustering system, how much disk I/O will be used in SPs?
Of course, it depends on my environment, but If you have some materials and benchmarks, please help me.

And in the case of search time,
if one user uses ad-hoc search on the search head, how much I/O will be used in SPs?

Thank you for your help.

[martin_mueller · Accepted solution]

The reference hardware recommendation applies for clustered Splunk instances as well: http://docs.splunk.com/Documentation/Splunk/6.2.0/Capacity/Referencehardware

Your 128kb/s suggests around 10GB/day, that's fairly low for three search peers so you will be good with lower IO performance from an indexing point of view. Three reference hardware search peers will be able to handle much much more than that.

Concerning how much IO an ad-hoc search uses... how long is a piece of string?
The slower your IO, the slower your IO-bound searches will be. See types of searches: docs.splunk.com/Documentation/Splunk/6.2.0/Capacity/HowsearchtypesaffectSplunkEnterpriseperformance

For further reading, there's a high-level overview in this docs book: http://docs.splunk.com/Documentation/Splunk/6.2.0/Capacity/IntroductiontocapacityplanningforSplunkEn...

[sunrise]

Thank you very much, martin_mueller.