Deployment Architecture

Retiring indexers

BrianAbbott
Explorer

I have an older cluster that is currently aging out. The peer group consists of three indexers which need to be dropped down to just one. RF=3 and SF=1 at this time.

I want to be sure that I am taking the correct steps to remove two of these indexers would be. Looking at https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Removepeerfrommasterlist it seems that I need only:

splunk remove cluster-peers -peers <guid>

Is that correct?

Will the SF be an issue or will it rebuild automatically?

Thanks.

Tags (1)

BrianAbbott
Explorer

I did set the RF to 1 (SF by default) because it esdbeen my understanding that the RF has to be set to 1 in order to accomplish what I am trying to do. Now that I reread it, perhaps I need(ed) to set the RF to be 2, as it had been 3.

https://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Takeapeeroffline#Take_a_peer_down_permane...
Because this version of splunk offline requires that the cluster return to a complete state before the peer can go down, certain preconditions are necessary before you can run this command:
• The cluster must have (replication factor + 1) number of peers, so that it can reallocate bucket copies to other peers as necessary and can continue to meet its replication factor after the peer goes down.

0 Karma

BrianAbbott
Explorer

I appreciate the follow up, quite helpful.

I did begin the process by putting one peer into a decommissioning status, where it has remained even until now. RF=1 SF=1. RF is met but one index is being a pain. It seems that one index has a bucket with no possible primaries. As such, I could delete the copy.

Is it acceptable to place the next indexer into decommissioning status even as the SF is technically not met?

0 Karma

tiagofbmm
Influencer

No, you need to wait for the decommission to finish. Let it keep running for longer and that should be fixed

0 Karma

tiagofbmm
Influencer

But wait, you changed the RF and SF before finishing the decommission?
Every bucket needs to have a primary. That's the premise for you to have all your data searchable
If your SF is 1 and it is not met, you'd need to later rebuild the tsidx files for that bucket later from the raw data (RF data)

0 Karma

tiagofbmm
Influencer

Go with splunk offline --enforce-counts. Used to remove a peer permanently from the cluster. Also known as the "enforce-counts offline" command.

http://docs.splunk.com/Documentation/Splunk/6.6.3
/Indexer/Takeapeeroffline.

it will get you the cluster in a valid state on the process. Remember you're moving to 1 Indexer only, you need to change the RF to 1 otherwise it won't be valid after you finish the decommission. SF will be fine as long as you bring the peers offline one at a time as the buckets will be copied to the last one standing, no need to rebuild or make them searchable again

After that you can remove the peer from the cluster as you mentioned

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...