FEATURE REQUEST: Rebalance buckets across new and existing indexers

[bandit]

I am hoping this will be added to a future release and can happen in real time and/or a maintenance mode for the indexers. Please UPVOTE if you agree so Splunk will prioritize this.

This seems partially implemented as of version 6.x http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Rebalancethecluster, however it has a major caveat and it is more of a relabeling of existing buckets and not a true re-balancing. This is definitely a nice feature but does not address the use cases below.

Note: Even though rebalancing occurs when a new peer joins the cluster, that peer won't participate in the rebalancing, because it does not yet have any bucket copies. The re-balancing takes place among any existing peers that have searchable bucket copies.

You could have to wait days or months and never have them re-balance forcing you to attempt to manually move buckets, trick index replication or prematurely delete historical events. In environments with limited disk space, this would have to be addressed immediately. I'd like to see re-balancing of the buckets whether or not the new indexers/peers have buckets or least have some manual option to do this.

I can think of at least 2 use cases for this. Please add any others you think may be relevant.
- New indexers have been added to a cluster for additional capacity, however they do not equally share the hosting of historical buckets.
- A high volume feed has been pinned to a single indexer resulting in higher disk use on an indexer.

Related threads: http://answers.splunk.com/answers/116072/how-do-you-balance-disk-usage-across-cluster-peers-after-ad...

http://answers.splunk.com/answers/113961/clustering-and-reassigning-primaries-issue.html

[davidpaper]

Delivered! As of 6.5:

http://docs.splunk.com/Documentation/Splunk/6.5.5/Indexer/Rebalancethecluster

Happy rebalancing.

[mics]

It is correct that version 6.5 has a new feature "Indexer Cluster Rebalancing". Unfortunately that feature is rebalancing the number of buckets and not taken the size of the bucket into consideration. Since bucket size can be very different then that is a very important factor. In an Enterprise solution with petabyte of data the size of buckets must be taken into consideration.

On .conf2016 one of our consultants talked to the developer of the "Indexer Cluster Rebalancing" feature and he was aware of the problem and was planning to look at it.

[hortonew]

This is now in 6.5 - http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Rebalancethecluster

[rbal_splunk]

This requirement is being tracked in Bug SPL-99206: Indexer Clustering Data Rebalance which is expected to be fixed in future major release.

[yannK]

You should also create an Enhancement Request case on the support portal.

[bandit]

Thanks for your suggestion, Yann. Enhancement case number 229757.