Deployment Architecture

moving indexes to new partition with rsync did not work

jericksonpf
Path Finder

Hi,
I recently had to move my hot/warm buckets for my splunk indexes to a new linux device on the same machine.
I use auto scaling on all my buckets
I used rsync -azv to copy over the hot/warm buckets from one directory to another so it looked like this:
rsync -azv /var/lib/splunk/index1/db /var2/lib/splunk/index1/

i made sure to change the ownership with chown -R splunk for the directory

I turned off splunk
i ran rsync on all the folders for the indexes i was moving
updated splunk-launch.conf to change $SPLUNKDB from /var/lib/splunk to /var2/lib/splunk
then i updated the index.conf files for etc/apps/search/local and for etc/system/local to $SPLUNK/index* (for each indexes)

When i started splunk up it immediately disabled all the indexes except _internal and one of the indexes that i had moved first as a test. The only way i could get splunk to work is to edit the indexes.conf file and have it make the home buckets in a new directory.
However the indexes in this new directory now have new hot buckets and now i can only search for events after the switch.

How do i reconcile these buckets. I want to get all the data from before the switch to be in the same folder. How do i do this without causing bucket naming collisions?

Tags (1)
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

It's a classic mistake, when using rsync, you have to exclude the hot buckets, to avoid duplicates bucket id when they rotate to warm.

see those posts :
http://splunk-base.splunk.com/answers/30986/why-is-my-index-disabled
http://splunk-base.splunk.com/answers/6114/whats-this-duplicate-bucket-in-my-index

To resume, identify the duplicates (one hot_X_, one db_X_X_ ) and remove the hot version.

View solution in original post

yannK
Splunk Employee
Splunk Employee

It's a classic mistake, when using rsync, you have to exclude the hot buckets, to avoid duplicates bucket id when they rotate to warm.

see those posts :
http://splunk-base.splunk.com/answers/30986/why-is-my-index-disabled
http://splunk-base.splunk.com/answers/6114/whats-this-duplicate-bucket-in-my-index

To resume, identify the duplicates (one hot_X_, one db_X_X_ ) and remove the hot version.

jericksonpf
Path Finder

This worked great, but duplicate warm buckets caused a conflict as well

0 Karma

kristian_kolb
Ultra Champion

Yep, you need to have a unique id for each bucket. That is the last set of digits. You can manually rename the directory names, as long as you an unused number. Don't mess with the epoch timestamps, though.

/K

0 Karma

kristian_kolb
Ultra Champion

you have a conflict with buckets 13 and 567 according to the errors. just change that value - not the 1366xxxxx parts.

See Yann's posts as well.

0 Karma

jericksonpf
Path Finder

Thanks for your response: how do i not mess with the epoch time stamps

0 Karma

jericksonpf
Path Finder

04-16-2013 19:33:27.165 -0700 ERROR DatabaseDirectoryManager - idx=pf_app_mobile bucket=db_1366089008_1365567758_13 Detected directory manually copied into its database, causing id conflicts [path1='/var2/splunk/lib/splunk/pf_app_mobile/db/hot_v1_13'

-0700 ERROR IndexProcessor - caught exception for idx=pf_systems during initialization: 'idx=pf_systems bucket=hot_v1_567 Detected directory manually copied into its database, causing id conflicts [path1='/var2/splunk/lib/splunk/pf_systems/db/db_1366114180_1366085206_567' path2='/var2/splunk/lib/splunk/pf_systems/db/hot_v1_567'].'

0 Karma

kristian_kolb
Ultra Champion

what did the error message say? STDERR and in splunkd.log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...