Installation

How do I migrate data from a single install (search head + indexer) to a search head + X indexers?

robertlynch2020
Motivator

Hi

I have worked out how to set up a distributed search on my beta site, but how do i move the current data "indexes" and "datamodels" off the search head to a new indexer?

Current = One search head + one indexer on the same install
Future = One search head + X indexers (New installs) + Original Indexer(With original data).

As all my APPs are installed on the current install, i think it is best if I keep this search head and move the data to a new indexer(and put in the old data), then add on X new empty indexers - Right?

So do I set up another Splunk install and copy over files from the original install to populate it? If so what files?

/splunk/var/lib/splunk, perhaps, is this folder or maybe a subset? [How come i cant find doc on this? i can't be the first person to do this right?]

Thanks in advance

Robbie

Labels (3)
1 Solution

woodcock
Esteemed Legend

First, go with indexer clustering so that you can do a rebalance of the old data. Copy $SPLUNK_HOME/etc/apps from the original server to the new Search Head. Setup the new Search Head with an outputs.conf to forward its logs to the Indexers. Disable local login to ALL Indexers. Initiate data rebalance on the Indexers from the Cluster Master.

BE SURE TO READ THE COMMENTARY BELOW THIS ANSWER!

View solution in original post

woodcock
Esteemed Legend

First, go with indexer clustering so that you can do a rebalance of the old data. Copy $SPLUNK_HOME/etc/apps from the original server to the new Search Head. Setup the new Search Head with an outputs.conf to forward its logs to the Indexers. Disable local login to ALL Indexers. Initiate data rebalance on the Indexers from the Cluster Master.

BE SURE TO READ THE COMMENTARY BELOW THIS ANSWER!

gpullis
Communicator

I'm doing the same thing, basically. Moving from a single Splunk server to a separate search head and indexer. I copied $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users to my new search head.

What's missing is my authentication config. What do I need to copy to get that?

woodcock
Esteemed Legend

It should be in authentication.conf and authorize.conf. If you used the GUI, this will be in $SPLUNK_HOME/etc/system/local/. You can see for sure by running this:

find $SPLUNK_HOME -name "auth*.conf"
0 Karma

gpullis
Communicator

Thanks! That got me my LDAP config.

I did have to manually re-enter the password for my LDAP binding account, but otherwise that's progress!Oh, and the fun part was trying to get the password saved faster than the search head could get the account locked out. 🙂

Now, how do I replicate local accounts?

Thanks again.

0 Karma

woodcock
Esteemed Legend

Local accounts should be in $SPLUNK_HOME/etc/passwd but if your new server did not get seeded with the splunk.secret file from the old server, then all the passwords will need to be reset once you copy this file.

0 Karma

gpullis
Communicator

Thanks. Lucky for me, I just have a few local accounts used for ops wall mount displays, and their passwords are documented. This would be a bummer for someone with a lot of local accounts.

If Splunk ever gets around to writing a guide for how to expand out of a single instance into multiple servers, splunk.secret would be a super important thing to mention early on.

The other thing is that, when you copy your $SPLUNK_HOME/etc/apps all your inputs come along for the ride, and you want to give some early thought as to where your inputs are running when the dust settles.

woodcock
Esteemed Legend

I agree, I should have mentioned it earlier.

0 Karma

rupeshn
Explorer

@woodcock

Similar type of situation, But my standalone indexer which acts as SH, Indexer and DS has very few(2-3) Apps. So I plan to delete those apps first, then set it as indexer and add 2 new indexers and add new SH. Would that be feasible? If so, what are Pros and cons involved. Please provide your inputs.

0 Karma

woodcock
Esteemed Legend

Tje very first thing that I would do in your situation is move the DS, if that is changing. The problem with that is that most people who do not know better use the CLI and set deploy-server to setup Deployment Clients instead of dropping a DeploymentClient app in $SPLUNK_HOME/etc/apps/. Make sure that when you update your DCs that you fix this mistake and use an app.

0 Karma

robertlynch2020
Motivator

HI

Cheers and thanks, i will give this a go. 🙂 This is how i was thinking i should give it a go.
It was the getting my apps over bit i was not 100% sure, but it makes sence that i can just copy them - cool cool cool.

Robbie

0 Karma

robertlynch2020
Motivator

One question.

Do i need to install the apps onto the indexers(or part of them), or will index clustering take care of index creation and datamodel creation on the new indexers?

Rob

0 Karma

woodcock
Esteemed Legend

Be sure to copy the splunk.secret file from the old server to the new servers BEFORE you start the new servers. That way, any encrypted passwords stored will still work. Generally it is safe to install everything everywhere so, yes, install the apps on the Indexers. Some of them will be necessary and some of them won't but the ones that aren't will be harmless.

0 Karma

robertlynch2020
Motivator

Cheers my man 🙂

0 Karma

woodcock
Esteemed Legend

Also, do not keep the original indexer if the new indexers are beefier in any significant way. Every search is only as fast as the WORST indexer. Having 99 blazing fast indexers and 1 slow one will be pretty much the same as having 100 slow indexers.

0 Karma

kml_uvce
Builder

move your current search head(etc/apps) to new search head , and change current (search head + indexer) as indexer

0 Karma

robertlynch2020
Motivator

Thanks for the answer 🙂

0 Karma

robertlynch2020
Motivator

Hi

I was looking at the doc and it looks like it should move my current install to an indexer and reimport my apps to a new search head.

SO I will try this and get back with updates

Rob

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...