Move three indexes to another volume/paritition

[riqbal47010]

we are facing the disk space in HQ site with almost all the indexers with 95% disk space is fully utilized.

Total disk space=10TB
Indexes = 7.5 TB
summary Index=1.5
500GB reserve for Splunk Operations.

now management approves extra 3TB for each Indexer.

my question is that:

can we add 3TB as another partition/volume and move(entirely) less expensive indexes to new volume ?

I need the detailed steps

[woodcock]

The easiest way is to use soft links. By default index instances exist here:

 $SPLUNK_HOME/var/lib/splunk/

So if you are going to rehost index foo on another volume say mounted at /splunk_slow_storage/ you would:

1: rsync $SPLUNK_HOME/var/lib/splunk/foo/* to  /splunk_slow_storage/foo/*
2: stop splunk
3: rsync one more time
4: rm -rf $SPLUNK_HOME/var/lib/splunk/foo/
5: rmdir $SPLUNK_HOME/var/lib/splunk/foo/
6: ln -fs /splunk_slow_storage/foo $SPLUNK_HOME/var/lib/splunk/
7: restart splunk

[riqbal47010]

Hi

In cluster environment I have two indexes named wineventlog and fortinet both indexes size are 1.5TB each.
we can move them to 3 TB Partition.

1- create a separate volume 3TB
2- update volume configuration in indexes.conf and roll out
3- check new volume permissions.
4- run rsync to copy the data.
5- put the CM in maintenance mode.
6- Stop the indexer-1.
7- run rsync again
8- perfrom step # 4 and 5 on all remaining indexers.
9- roll the configuration
10- start the indexers one by one.
11- remove the CM from maintenance mode.

Please correct me if i am wrong.

[gcusello]

Hi @riqbal47010,
if you can you could enlarge the data partition.
Otherwise you could move some indexes in the new partition.
to do this you have to follow the steps:

• analyze indexes grogth executing a capacity planning;
• identify the indexes to move;
• stop splunk;
• edit indexes.conf changing some indexes path to the new location;
• move the folders of the choosen destination;
• restart Splunk.

Ciao.
Giuseppe

[riqbal47010]

Is that really simple... ?
amazing and little confusing.

However I identified one index which is index=cisco_asa i
its total size is 3.5T
2.2T colddb
1.1T datamodel_summary
1.3T db
12K summary

4.0K thaweddb

Now is that possible that
I create a separate volume 3TB

1- put the Indexer cluster in maintenance mode
2- update new volume stanzas in indexes.conf on CM
3- make sure that Splunk is able to create directories/files in new mount point.
4- roll the configuration
5- Now splunk should write the cisco_asa cold date to new volume/mountpoint

Please correct me if i am wrong.

[gcusello]

Hi @riqbal47010,
in your question, you forgot to say that you have an Indexer Cluster!

In theory you could follow the same approach moving configurations (indexes.conf) on Master Node and data (bucket's folders) in peers working with Splunk down in all servers; but I'm not sure that data consistency will be maintained because there's a cluster and I'm not sure about the moving of replicated buckets.

In addition, if you have 3.5TB in the cisco_asa index, how can you move it in a storege of 3 TB?

Sorry but probably the best idea it's to ask to Splunk Support!

Ciao.
Giuseppe

[riqbal47010]

initially my plan is to move colddb to 3 TB partition.
I hope no problem will pop up.

[riqbal47010]

OR I have two other indexes named wineventlog and fortinet both indexes size are 1.5TB each.
we can move them to 3 TB Partition.
Whats your advise.