Adding indexer cluster peers with different volume structures

[cdoebert]

We're in the process of upgrading our indexer cluster, with the plan ultimately being to phase out the old indexer cluster. At issue is that the new indexers have a mix of SSD's and HDD's that are on two separate volumes, and the old indexers only have one HDD volume.

We had planned to use the "cluster fixup" method to do this migration, wherein we add the new indexers as cluster peers to the existing cluster, fix up the entire environment to replicate buckets onto the new indexers, take out one of the old indexers from the cluster, fix up the cluster again to replicate more buckets onto the new indexers, rinse and repeat until only the new indexers remain.

The problem with this method lies with a disparity in where the indexed data will be stored between the old and new indexers. Since the new indexers have two volumes (that we will call /data/ssd and /data/hdd) and the old indexers only have one volume (that we will call /data), the homePath and coldPath for the indexes will be different within the indexer cluster.

Example:
OLD HOME PATH: /data/some_index/db
NEW HOME PATH: /data/ssd/some_index/db
OLD COLD PATH: /data/some_index/colddb
NEW COLD PATH: /data/hdd/some_index/colddb

How, if at all, can I have the cluster handle this?

Some preliminary ideas:
- Have a local indexes.conf on the new indexers that explicitly defines every index path during the migration and overrides what the CM distributes
- Take a (big) outage on the cluster and manually move all the data within the old indexers to mimic the new structure (e.g. mv /data/some_index/db /data/ssd/some_index/db)

[micahkemp · Accepted solution]

You can indeed configure different home/cold paths in etc/apps/<someapp>/local (please not in system/local) to accomplish that. I wouldn't consider it invalid, it's just annoying to manage.

Once you have things fully migrated you should work to moving your CM-deployed configuration to what it needs to be for the new indexers, and remove the manually installed configuration.

[cdoebert]

I've run into a problem with this, and the problem checks out in the docs.

http://docs.splunk.com/Documentation/Splunk/6.5.3/Admin/Wheretofindtheconfigurationfiles
(still holds true in the latest version, but I'm running 6.5.3)

Even when I'm a bad admin and I put this in system/local, I was still overridden by the CM, and this is why:

1. Slave-app local directories (cluster peers only) -- highest priority
2. System local directory
3. App local directories

So now my sub-question is "can I put an app in slave-apps/ and it not be squashed by the CM?" I suspect the answer is "yes, until you push a new bundle out", but I'm not sure.

[micahkemp]

What if you configure the CM to push indexes.conf using $SPLUNK_DB, and configure that appropriately on each indexer via splunk-launch.conf?

I would probably shy away from manually placing things in slave-apps.

I apologize for the incorrect answer, and the above is certainly just a “maybe try this” instead of an actual answer.

[micahkemp]

Wait, that says slave-apps/local is higher than system/local. I had assumed your indexes app was using master-apps/app/default/indexes.conf, which you could override with apps/app/local on the indexer.

So that may be an option. Move your indexes.conf on your CM to master-apps/app/default/indexes.conf.

[cdoebert]

That did the trick! Once I moved the cluster indexes.conf from local to default, the local changes took.

Thanks Micah!