Deployment Architecture

Is it possible to bring new servers online into the respective pools and have them sync in such a way that we can remove the old servers one at a time?

trinity1571
New Member

Would it be possible to bring the new servers online into the respective pools and have them sync in such a way that we can then remove the old servers one at a time. If we can, which is the best order in which do this?

What would be the best way to go about this transition?

0 Karma
1 Solution

cpetterborg
SplunkTrust
SplunkTrust

When we add new indexers, we add them in, rebalance, then take the old ones out and rebalance again. It hasn't failed us yet. Do you need details on doing something like this?

View solution in original post

cpetterborg
SplunkTrust
SplunkTrust

When we add new indexers, we add them in, rebalance, then take the old ones out and rebalance again. It hasn't failed us yet. Do you need details on doing something like this?

trinity1571
New Member

Details would be wonderful, thank you.

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Rebalancing

From the docs, on the cluster master:

splunk rebalance cluster-data -action start [-index index_name] [-max_runtime interval_in_minutes]

Which I just do:

splunk rebalance cluster-data -action start

because I want all the indexes rebalanced and the default runtime is usually enough. This may take a while, but in most cases where we are doing this, it takes less than 30 minutes.

Removing the indexer (search peer)

Caution: The following are from notes that I have in my administration notes. I'm pretty sure they are correct, but if not, I'm not going to take any responsibility for problems that may arise. I have not had any bad effects on my environment, so I think you should be safe.

Take the indexer offline by running the following on the indexer to be removed:

splunk offline --enforce-counts

Then you will have to wait for the status on the CM (cluster master) to show that all is well through the Monitoring Console. All data should be searchable, the search factor met and the replication factor met across all indexers.

To see the status of the cluster and get the for the next step, on the CM run:

splunk show cluster-status

Run this command on the CM for the Indexer you are decommissioning, once it is finally offline. This will keep it from joining the cluster if it comes back on line.

splunk remove cluster-peers -peers <guid>

Only remove one search peer at a time, then make sure the cluster is all okay (all the indexes are fully replicated and searchable) before you remove a second one.

So the steps are:

  1. Add in all new indexers
  2. Rebalance
  3. Remove one search peer
  4. Ensure good indexer cluster status
  5. Rebalance
  6. Repeat steps 3-5 until you have removed all the search peers that you need to remove
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...