Deployment Architecture

Migrating deployment server - check my work please :)

paimonsoror
Builder

Hi all;

I have to decomission our current deployment server (forwarder configs) and was hoping to get an extra set of eyes on the work that I plan on doing:

  1. INSTALL Splunk on new server
  2. COPY server classes to new server
  3. COPY ALL deployment-apps over to new server
  4. UPDATE the app with the client config on the new server to point to the new server
  5. UPDATE the app with the client config on the old server to point to the new server
  6. RELOAD the old deploy server

My expectation here is that the reload will re-push the updated deploymentclient.conf file out to all the servers, which will in turn restart the forwarders and make them connect to the new server.

paimonsoror
Builder

Wanted to update everyone on what I ended up doing :).

  • So the first thing i did, was on the original Deployment Server, I created a new deployment app called 'enterprise_deploymentclient' that had a deploymentconfig.conf file that pointed to the new server

  • I then copied the deployment-apps folder to the new server exactly.

  • After that, I copied over the serverclasses.conf to the new server.

  • New server was then recycled.

  • Next, i worked through each server class, removed the original deploymentclient 'app', and replaced it with the new one

  • Saved the class, and then verified that the servers soon connected to the new server

  • Rinsed and repeated for each class, and went page by page

In my opinion, this was the "safest" way of doing it vs just updating the original deploymentclient.conf file and pushing it out in one fell swoop.

0 Karma

tmarlette
Motivator

So migrating a DS can be a bit tricky sometimes, but ill give you some steps I've used in the past.

on the old DS
1. make a full backup of /opt/splunk/etc/* on the old DS
2. tar up the following things independantly
a. deployment apps > deployment_apps.tgz
b. system / local > local.tgz
c. apps (I use sym links from /deployment-apps to /apps on the DS to maintain up-to-date apps in a single repo. If not, you won't
need apps)

This 'should' give you the base config of your old DS. Should being operative due to editing default directories anywhere.

  1. install Splunk on the new machine.
  2. shut down splunk
  3. delete /deployment apps from new server
  4. delete /system/local from new server
  5. un tar deployment_apps.tgz to /opt/splunk/etc
  6. un tar local.tgz to /opt/splunk/etc/system
  7. start splunk

before doing this make sure you have that full backup, because if there's ever anything you need for config, it's going to be in /opt/splunk/etc

0 Karma

paimonsoror
Builder

Thanks @tmarlette! I think one catch that we would need to worry about is making sure that the app on the DS that has deploymentconfig.conf is updated with the new DS right? Else when we now go and update the existing forwarders to point to the new DS, it will keep bouncing back and forth

FYI: we deploy the deploymentconfig.conf as an app vs configuring it during forwarder install

0 Karma

tmarlette
Motivator

do you mean, deploymentclient.conf? I'm not familiar with the other file.

assuming you mean deploymentlcient.conf here:

so... if you are using an IP address in your deploymentclient.conf, the app your using will be useless, because as soon as you change the IP address on your DS, your forwarders lose connection. UNLESS you're going to use the same IP address, which can sometimes prove tricky.

The workaround:
use a DNS address like 'splunkds.mydoman.com' and stick that in the /etc/system/local folder of all of your forwarders. That way you can change the DNS mapping to a different IP it's basically a hot cut.

0 Karma

ddrillic
Ultra Champion

I have some doubts about -

-- My expectation here is that the reload will re-push the updated deploymentclient.conf file out to all the servers

0 Karma

adonio
Ultra Champion

when you say: "copy server classes" i assume you mean that you will copy serverclass.conf

0 Karma

paimonsoror
Builder

Thats correct 🙂 thanks for the clarification @adonio

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...