What is the process of decommissioning indexers?

[vrmandadi]

When you run the offline command permanently on an indexer.

1) How much time does it take to reassign the data to other members in cluster
2)Can we run offline command on three indexers at a time or do we need to wait for anything?
3) Adding and removing members is correct order or the other way around?

[nickhills · Accepted solution]

1.) It depends - how much data your indexer hold primary copies for, network/disk IO, cpu load etc. (It can take some time)
2.) No - Dont do this, decom one indexer at a time.
3.) Add new indexers first - more indexers in your cluster will speed up the decom process for the old ones, and will begin balancing buckets to the new peers as soon as the process starts.
Once you have added all your new indexers, remove the old ones one at a time.

[vrmandadi]

when you say some time like 6TB storage ,does it take days? or hours.We have a lot of issues with disk space space falling below 5000MB and thus the indexer stopped accepting new data.This made the HF queues getting filled.All of sudden the data burts in , data delay ...Does this cause data loss even if other Indexers are running?

[nickhills]

Add your new indexers, then update your forwarders to send data to the new indexers (not the old ones)
This will prevent the disk space on the old indexers causing the forwarding queues to fill up.

Your first decomissioned idx will have to transfer (ideally) primacy of approx 33% of buckets, and once that is done, the cluster will replicate buckets to the remaining 5 indexers.
With 6TB volumes, this is likely to take several hours (possibly days depending on spec etc).

Note - you could start by doing a data rebalance which will split the data equally between the old and new peers - whilst this would speed up each old decom, it is a very lengthy process, and would probably take longer than just running the decom process.
By the time you have decomissioned all 3 old indexers, you data should be reasonably well balanced across the new servers.

[vrmandadi]

Thank you

[vrmandadi]

@nickhillscpl how do we know that that a indexer which is decommissioning has transferred all its data to the other cluster members