Getting Data In

How do I migrate Indexes from one multisite cluster to another multisite cluster?

harsmarvania57
SplunkTrust
SplunkTrust

Hi Community Members,

I would like to migrate indexes from one multisite cluster to another multisite cluster. Both the multisite cluster have same RF/SF. What will be the steps for this ?

I have gone through few of the answers on community and try to find out documentation for multisite index migration but no luck.

My understanding is
1.) Copy buckets from db and colddb directories from old multisite cluster to new multisite cluster
2.) Check any bucket ID conflict and if so rename those bucket with newer ID

Now questions are
1.) Do I need to copy buckets starting with rb_timestamp_timestamp_ID_GUID ?
2.) Do I need to copy <index name>.dat file ?

Thanks,
Harshil

0 Karma
1 Solution

dxu_splunk
Splunk Employee
Splunk Employee

1) make sure theres no GUID conflicts between the clusters - otherwise if we move buckets from one cluster to another that has indexers with the same GUID, it could conflict with an existing bucket.

2) do the clusters use the same available_sites and site namings?
if not, we'll need to configure to use the site_mappings settings before we move buckets from one multisite to another multisite. please see https://docs.splunk.com/Documentation/Splunk/7.1.1/Indexer/Decommissionasite#Syntax
(the documentation is in decommissioning a site, but the concept is similar - buckets that originated from siteA that are no longer available need to get a new "originSite", because it has to satisfy the originSite policy - we cant get any copies on siteA anymore so we'll map it to a new site)

3) when copying over the data, you can copy just the db_ versions of the bucket, but it'll probably be faster (and more correct) to copy over everything (db_ and rb_) of the index, (*more correct because its possible a bucket only has rb_ copies and no db_ copy)

View solution in original post

dxu_splunk
Splunk Employee
Splunk Employee

1) make sure theres no GUID conflicts between the clusters - otherwise if we move buckets from one cluster to another that has indexers with the same GUID, it could conflict with an existing bucket.

2) do the clusters use the same available_sites and site namings?
if not, we'll need to configure to use the site_mappings settings before we move buckets from one multisite to another multisite. please see https://docs.splunk.com/Documentation/Splunk/7.1.1/Indexer/Decommissionasite#Syntax
(the documentation is in decommissioning a site, but the concept is similar - buckets that originated from siteA that are no longer available need to get a new "originSite", because it has to satisfy the originSite policy - we cant get any copies on siteA anymore so we'll map it to a new site)

3) when copying over the data, you can copy just the db_ versions of the bucket, but it'll probably be faster (and more correct) to copy over everything (db_ and rb_) of the index, (*more correct because its possible a bucket only has rb_ copies and no db_ copy)

harsmarvania57
SplunkTrust
SplunkTrust

Hi @dxu [Splunk],

Thanks for your valuable time.

Yes, both clusters use the same available_sites and site namings.

0 Karma

vadivel_parames
Explorer

Hi, @harsmarvania57, i have a similar requirement. Can you provide me the steps how to migrate the date from one multi-site index cluster environment to an another multi-site environment? I have 2 sites, each site has 3 peers (total six peers) in one environment and the same configuration in an another environment. Should I copy buckets from all the indexers in the old environment to all the indexers in the new environment?

0 Karma

adonio
Ultra Champion

hello there,

before going any further with this, do you have to move the indexes?
what is the goal of this data move? what purpose does it serves?
imho there are almost always better ways to achieve of the goals behind the need for moving data (indexes) especially clustered ones, without the pain of moving the data.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @adonio,

Yes, I would like to move the indexes. I am in process to move application data from 1st multisite cluster to 2nd multisite cluster due to security reason. And once application data will be separated from 1st multisite cluster, both the clusters will not have any relationship. Even data will be searched from different SHC and those SHC will search data from their respective multisite indexer cluster.

I know that to move data in clustered environment is bit tricky but I am still looking for good solution if anyone have.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...