Remove Site from Indexer Cluster

[stepheneardley]

I've just finished adding new physical indexers to our existing multi-site indexer cluster and I'm trying to figure out the safest method for removing the old virtual indexers.

We started off with a 2 site cluster with each site having 3 members and the following config.

available_sites = site1,site2
multisite = true
replication_factor = 2
site_replication_factor = origin:1,total:2
site_search_factor = origin:1,total:2

When I added the new indexers I created 2 new sites (3 and 4) and amended the config as follows.

available_sites = site1,site2,site3,site4
multisite = true
replication_factor = 4
site_replication_factor = origin:1,total:4
site_search_factor = origin:1,total:4

Now that the upgraded cluster has equalized I'm trying to figure out what the safest method for removing sites 1 and 2 is.  I think it should be;

1. splunk offline --enforce-counts (while watching the indexer clustering dashboard on the CM waiting for all data to be searchable before offlining the next).

2. Put the cluster into maintenance mode and update server.conf on the CM as follows;

available_sites = site3,site4
multisite = true
replication_factor = 2
site_replication_factor = origin:1,total:2
site_search_factor = origin:1,total:2

3. Disabled maintenance mode.

Any and all thoughts/past experiences appreciated.

[garyjohnson48]

Before you take them offline, change your available sites to site3, site4 so that the CM is operating only on those sites.  Run splunk offline for 24hrs on site1 and site2 before --enforce-counts to make sure everything is running smooth with your new setup. 

Your single-site replication factor shouldn't drop below the lowest amount of indexers you have on a particular site so instead of dropping the replication factor to 2, set it as 3 since that is the number of indexers you have in the site. If you set it as two you will get errors.