Upgrade from distributed to clustered environment retaining configurations and data?

[Skins]

Hi ,

Is there a way to upgrade a distributed environment consisting of 1 x SH, 2 x IDX and a DS to a HA clustered env consisting of

3 x SH, 1 X Deployer, 3 X IDX and a Cluster Node ? (assume cluster node also LM)

with an aim to keep data already in the distributed indexes

is this possible and which steps should be carried out in what order ?

gratzi.

[Skins]

Thanks for the detailed breakdown - v.helpful.

I have read the section on migrating current indexes to a cluster and that seems the way we will do things - probably not add the 3rd indexer and see what the performance is like as is - we can always add to the cluster and rebalance at a later date.

with regards to migrating the SH - having read the following doc it all seems straight forward building a new SH cluster and migrate apps to the Deployer.
https://docs.splunk.com/Documentation/Splunk/6.6.3/DistSearch/Migratefromstandalonesearchheads

The only other Q i have is preferred order - build the SH cluster and connect this to the distributed indexers and then migrate to a cluster ?

[traxxasbreaker · Accepted solution]

I'm going to assume that you are going to be building new indexers rather than clustering your existing ones. If that is the case, you would need to start by building your cluster master and new indexers.

For your search heads you will need to setup the new search head cluster with the deployer and get it searching both the new indexers and the old indexers, assuming your current 2 aren't going to become part of your cluster. It's especially important to move the search heads first if you are also upgrading Splunk versions on this move since the search heads have to be either the same or a higher version than the indexers. Then, you'll want to copy over the default app directories from your old search head to the deployer and push them out, but be very careful not to include out of the box apps like "search" in that push. The local app directories are best setup by copying them to each cluster member so that users will be able to delete their own content. If you push them from the deployer, the cluster members will see them merged into default from the deployer, so users wouldn't be able to delete their own content and some don't like that.

For the user level directories on the search head, I just copied them to each SHC member because we're using LDAP, but if you have locally defined users you'd likely want to treat that migration similarly to the app migration in which you define default content on the deployer and local on each cluster member.

Next, make sure your new indexers are setup with the same input and parsing configurations as your old ones. Then, you have 2 options, and the best one for you will depend on your comfort level and the tolerance for a complete outage of indexers in your environment.

Option 1:
Have your forwarders start sending new data to your new indexers. After you flip your forwarders, take a good look at your old indexers to make sure they are not still taking in data. Once you're sure they aren't, restart all of them to roll any hot buckets to warm. Then you should be able to use the guide at https://docs.splunk.com/Documentation/Splunk/7.0.0/Installation/MigrateaSplunkinstance to actually move the data. Note that the legacy buckets won't replicate, if that is a huge concern you might want to consult with professional services since they do have some methods of making those buckets replicate anyway. As the data is activated on the new servers, you should take down the old servers that the data came from to prevent duplicates from being returned in the search results.

Option 2:
Take your old indexers offline, follow the migration instructions linked above, bring up your new indexers, then have your forwarders redirect to the new ones. This is simpler to implement but results in more downtime. You could theoretically reduce the downtime by doing an rsync in advance, then a second rsync during the change window just for the updates.

After that, since you are going from 2 indexers to 3, you would want to rebalance your cluster so that the buckets are being searched evenly across the indexers. Otherwise the 2 that got the migrated data would be getting most of the load.

Now, if you are looking to migrate your existing indexers into a cluster rather than building new ones, you'll want to check https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Migratenon-clusteredindexerstoaclusterede... for guidelines on how to do that. You'll still need to address your search heads first if you are upgrading to a newer version.