Hi everyone,
I am working on Splunk clustered environment, where i have 3 indexers,1 search head, and 1 head.
Now i am facing the problem with low disk space. i configured index in index.conf in head node.
my index is replicating in 3 indexers. if want to change index path in every indexer. what should i do please help me on this. now i have given path D://splunkdb/indexname. my index resides on D drive in every indexer. now i want to move this to F: drive in each indexer.
I would not suggest making any changes on a Indexer locally. Here is my suggestion:
Lets assume that the original Index is at /opt/splunk/var/lib/splunk/defaultdb
, and the new location will be at /splunk/defaultdb
.
In order to limit the down-time of each indexer to the minimum, we will do it in a few steps. First, while the service is still running, rsync
the data from the old location to the new one:
Note: When running the rsync
command, use a trailing slash only after the source path, and not after the destination path.
rsync -auv /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb
.
This will create an initial copy of the data to the new location. The initial sync may take some time, depending on the size of your data, there also may be a lot of changes to the data when that process is done due to bucket rolling from hot/warm/cold.
Now we want to do another rsync
to send the recent changes, this will be a lot faster. But now we will add the --delete
argument, so it deletes rolled buckets from the new location. like so:
rsync -auvv --delete /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb
if you want to be able to look at what rsync
did, you can send output to a log file by adding the --log-file=/tmp/rsync-`date %s`.out
to the command.
Put the CM in maintenance mode and stop Splunk, and do the final sync.
On the master:
$SPLUNK_HOME/bin/splunk enable maintenance-mode --answer-yes
On the Indexer:
$SPLUNK_HOME/bin/splunk stop
Do the final sync:
rsync -auvv --delete /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb
Move away the old index data to a backup location:
mv /opt/splunk/var/lib/splunk/defaultdb /opt/splunk/var/lib/splunk/OLD.defaultdb
Latstly, create a symlink from the new location to the old one:
ln -s /splunk/defaultdb /opt/splunk/var/lib/splunk/defaultdb
.
Now you can start splunk, and not have to mess around with indexes.conf
After you start the indexer, make sure that the buckets are visible by check the RF/SF on the CM, after that you can take the cluster out of maintenance mode, to fill in the few buckets that been rolled while that indexer was down.
$SPLUNK_HOME/bin/splunk disable maintenance-mode --answer-yes
Repeat the same process 1-7 on each indexer in the cluster, one indexer at a time.
After doing this on all indexers, you can change the path in indexes.conf
on the master, and push out the new bundle.
A Indexer restart is required when changing the path of a index, so the master will initiate a restart.
After you did all that, and have confirmed that all the buckets are visible in Splunk, you can remove the old data, and delete the symlink:
rm -rf /opt/splunk/var/lib/splunk/OLD.defaultdb
rm /opt/splunk/var/lib/splunk/defaultdb
Please comment if I missed something.
I hope this helps.
Thanks a lot. Minor typo - step 7 should be "disable maintenance-mode", not enable.
Is there a reason you disable and enable maintenance-mode between every Indexer change? Can't you keep it on and change them all and then disable?
Thanks for making me aware of the typo. Fixed.
The reason for doing it one at a time, is to minimize the downtime, and a massive cluster-wide bucket fix-up. Taking your time moving them over ensures that the replication factor will remain reasonable and searches will be able to continue, since everyone has a different RF/SF setup.
Enabling and disabling maintenance mode will fix up missing buckets after the data move.
Hi,
To migrate indexes in a cluster configuration, you can proceed as follows:
TO MIGRATE SPECIFIC INDEXES: (modification of indexes.conf)
For each peer node of your cluster, one by one, migrate your data:
In master node:
Apply the bundle configuration:
$SPLUNK_HOME/bin/splunk apply cluster-bundle
This will achieve a reload of the cluster without restart, in splunkd.log of master node you will see a message like:
INFO CMMaster - All peers have reloaded the bundle without a restart
Verify your cluster is fully synchronized in master dashboard
Verify your data
TO MIGRATE ALL INDEXES IN PEER NODES MODIFYING THE $SPLUNK_DB:
For each peer node of your cluster, one by one, migrate your data:
The global modification of $SPLUNK_DB is easier as you don't have to alter the bundle configuration, but this does not allow to selectively migrate data by index.
These operations are not destructive but should be carefully proceeded, if possible they should be qualified in testing environment as for any configuration change, and off course you should have up to date backup of your data.
Regards,
Guilhem
Thank you for this well written answer.
We are going to try this soon on our cluster, since we will separate cold from hot/warm buckets on different partitions and we want to keep the cluster active during the move. (I mean that is why we have clusters right?)
I will comment here afterwards to tell you how it went.
Hi,
You can move all indexes from one place to another by changing the SPLUNK_DB env variable in splunk-launch.conf, see:
http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/Moveanindex
If you want to change only the location of a specific index, you need to modify the bundle configuration from your master node and apply it to your peers
See my new answer
if i change the bundle configuration from master node,what about the data indexed already.
yes exactly..i have clustered environment.
You're using some mixed terminology here. Can you clarify whether you have a clustered index environment?