We have a 2 site multisite cluster with the following cluster configuration.
The cluster contains 30 indexers total, 15 at each site. There is over a petabyte of data stored across the two sites.
[clustering]
cluster_label = StarfishCluster
mode = master
multisite = true
replication_factor = 2
search_factor = 2
available_sites = site1,site2
site_replication_factor = origin:1,site1:1,site2:1,total:2
site_search_factor = origin:1,site1:1,site2:1,total:2
We may have to move the servers that are in site 1 from one datacenter to another data center. The data center is several hundred miles away so these servers will be offline for over a week. How can we safely take down a site, then re-enable that site at a later time without losing data or having Splunk encounter issues with ingest. As a test we have put the cluster in maintenance mode then taken down all of the hosts within a single site but Splunk stopped indexing data in the other site which remained online. We also experienced an increase in resources which was expected.
Is there documentation available on exactly how to safely take down a site without impacting indexing and search availability?
