Installation

How to install upgrade from version 5.0.1 & complete migration with complicated Splunk configuration?

manderson7
Contributor

I've searched and seen a few migration threads, but my setup is a bit strange.

Two physical servers, 1st running splunk search head in /opt/splunk_sh, a splunk indexer in /opt/splunk_ind, and the deploy server under /opt/splunk_ds. The second physical server is running a splunk indexer in /opt/splunk_ind. All are running version 5.0.1.
I'd like to move onto 6.3.0, and so I've setup a VM for that purpose, and the plan is to move the search head function to the new vm. I'm now concerned about my order of operations:
1. stop splunk_sh on old server 1
2. copy /opt/splunk_sh to new server as /opt/splunk
3. install splunk6.3 on new server
4. startup new server

Once that's satisfied, upgrade the splunk indexers to 6.3?
Also, I need to move the deployment server off the old server to new vm, as well as license server. Uncertain about that order of operations as well.

Any advice is welcome here.

Labels (4)
0 Karma
1 Solution

woodcock
Esteemed Legend

First of all, there is no reason to run a separate Splunk instance for your Search Head, Indexer, and Deployment Server on server 1; you should merge your configuration files and run a single instance on server 1 which will greatly uncomplicate things. In any case, much of this is arbitrary (can be shuffled) but this is how I would do it and it will definitely work:

SERVER 1:
splunk_ds/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_sh/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

SERVER 2:
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

NEW VM:
copy /opt/splunk_sh/* to new server as /opt/splunk/
MANUALLY (*know* what config files are important) merge /opt/splunk_ds/ intto /opt/splunk/.
upgrade to new version (copy new files over old files in same directory)
Install license keys here
Upgrade your deployment_client App (deploymentclient.conf) to point to this DS
Upgrade your deployment client App ([license] stanza with the master_uri setting in server.conf) to point to this LS
/opt/splunk/bin/splunk start # This is now Search Head and License Server and Deployment Server

SERVER 1:
remove /opt/splunk_sh/ and /opt/splunk_ds/ # already migrated
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

SERVER 2:
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

View solution in original post

woodcock
Esteemed Legend

First of all, there is no reason to run a separate Splunk instance for your Search Head, Indexer, and Deployment Server on server 1; you should merge your configuration files and run a single instance on server 1 which will greatly uncomplicate things. In any case, much of this is arbitrary (can be shuffled) but this is how I would do it and it will definitely work:

SERVER 1:
splunk_ds/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_sh/bin/splunk stop # tar up the directory and store it somehwere, just in case
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

SERVER 2:
splunk_ind/bin/splunk stop # tar up the directory and store it somehwere, just in case

NEW VM:
copy /opt/splunk_sh/* to new server as /opt/splunk/
MANUALLY (*know* what config files are important) merge /opt/splunk_ds/ intto /opt/splunk/.
upgrade to new version (copy new files over old files in same directory)
Install license keys here
Upgrade your deployment_client App (deploymentclient.conf) to point to this DS
Upgrade your deployment client App ([license] stanza with the master_uri setting in server.conf) to point to this LS
/opt/splunk/bin/splunk start # This is now Search Head and License Server and Deployment Server

SERVER 1:
remove /opt/splunk_sh/ and /opt/splunk_ds/ # already migrated
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

SERVER 2:
upgrade to new version (copy new files over old files in same directory)
Modify /opt/splunk/etc/system/local/deploymentclient.conf to point to new DS
Modify /opt/splunk/etc/system/local/server.conf to point to new LM
/opt/splunk/bin/splunk start # Indexer now started

manderson7
Contributor

Created the deployment app, deployed it to a client, but it's not taking precedence over the file in /opt/splunkforwarder/etc/system/local/deploymentclient.conf.

0 Karma

manderson7
Contributor

NM on that, I'm having firewall issues...

0 Karma

manderson7
Contributor

I dig it. Yeah, running each splunk type as a different instance on the servers really threw me the first time I saw it. not a fan.

My current plan, please comment if you feel like it 🙂
1. Setup new license/deployment vm
2. Test dev or ua box w/ new deploy server (create app w/ new deploymentclient.conf file to disable current file, pointing to deploy/license server), an aside: do the universal forwarders need the license information, or am I including that possibly in this deploy app for the search peers?
3. on legser1: splunk_sh/bin/splunk stop # tar up the directory and store it somehwere, just in case
4. also on legser1: splunk_ds/bin/splunk stop # tar up the directory and store it somehwere, just in case
5. copy data (easy or manual) from legser1:/opt/splunk_sh/ to newser1:/opt/splunk
6. install splunk 6.3 on newser1
7. spin up newser1, login and poke around to see what's broken?
8. once known healthy, continue with indexer upgrades?

0 Karma

woodcock
Esteemed Legend

The only things that need to be pointed to the license server are the indexers but it is harmless to send the configuration stanza it to all servers (and it may make things simpler to package it that way).

0 Karma

manderson7
Contributor

Thank you for the advice, and for helping calm my nerves on this seemingly massive project.

0 Karma

woodcock
Esteemed Legend

So you are upgraded now?

0 Karma

amiracle
Splunk Employee
Splunk Employee

That sounds about right, and you can either setup a new license master and just add the indexer(s) to it as well as the search head(s). I don't think that it matters how you copy over the deployment server (before or after the license master). Additionally, you can just setup the new deployment server on the new box and just copy over the configs once the new box is up and running in the VM.

manderson7
Contributor

I'm trying to avoid as much downtime as possible. I've found that the license master is running on the primary search head instance. I figure once I bring down the primary search head, the splunk installation is useless until I bring up
1. a license server
2. a search head

Is that accurate?
If that's the case, I'll need to have a license server ready, and have my license keys ready to import, as well as have my search head vm ready for duty.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...