Knowledge Management

[SmartStore] Migration to Smart Store and looking for some details around this process?

rbal_splunk
Splunk Employee
Splunk Employee

We are planning to enable Smart Store (whether for all indexes or for just some indexes), our cluster will go through a migration process that needs to get all it's (s2 enabled) buckets into the remote storage. We have looked at the documentation and were looking at some additional information around this process.

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Here is outline of the Process:

1)when splunk indexers first start up with Smart Store enabled, it will check for the existence of a file ".buckets_synced_to_remote_storage" in each Smart Store enabled index's homePath

2)if there is no such file, we will start the migration process for that index - every bucket that is found in the index will be added to a list of pending upload buckets

3)this list is saved into a file in var/run/splunk/cachemanager_upload.json, so that if Splunk crashes before all the uploads finish, this file will be used to resume the migration+upload process on the next startup.

4)Splunk will create the .buckets_synced_to_remote_storage file in that index, even though nothing might be uploaded yet. We will use the cachemanager_upload.json file from now on to track needed uploads.

5)Splunk will start normally now, but in the background the indexers will upload its buckets in the pending upload list to the remote storage, following this order:

i)Splunk will first upload buckets that was created locally - by matching the buckets' guid with its own guid. So in migration, indexers will first upload its own created buckets into remote storage

ii)After its local buckets are uploaded, Splunk will then try to upload other indexers' buckets - but first it'll check the remote storage if that bucket already exists. There's a good chance the other indexers have already done this upload, so we will skip uploading for buckets that already exist in the remote store.

At the end of this migration process, all Smart Store enabled indexes should have their buckets safely persisted in the remote storage, and there will be a .buckets_synced_to_remote_storage file marker in that index for all indexers, as well as an empty cachemanager_upload.json file!

0 Karma

aakashbhalla1
Engager

Suppose we are migrating a non-smartstore indexed data to smartstore. Data exists in both hot/warm and cold locations. After the s3 upload process is complete, how will we validate that the local copies of the data is deleted?

0 Karma

srajarat2
Path Finder

The local data is not evicted by default after the migration. You can validate them by checking by running |rest /services/admin/cacheman | dedup title and verify the buckets, its status, size etc.,

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...