Deployment Architecture

Force search head cluster to forget old nonexistent members without performing remove shcluster-member command

m_efremov
Explorer

Some time ago we had a Splunk search head cluster with nodes named “vm-splunk-sh01”, “vm-splunk-sh02”, “vm-splunk-sh03”. Then we added some new nodes, i.e. vm-splunk-nsh01, vm-splunk-nsh02 etc. After successful adding of these new nodes to cluster we stopped old nodes vm-splunk-sh01, vm-splunk-sh02, vm-splunk-sh03 without removing them by command “splunk remove shcluster-member” and re-use these machines by clearing and reinstalling them (it’s my mistake). Now there are following strings in the new nodes’ logs:

2017-06-20T13:06:53.880Z I REPL [ReplicationExecutor] Error in heartbeat request to vm-splunk-sh02:8191; Location18915 Failed attempt to connect to vm-splunk-sh02:8191; couldn't initialize connection to host vm-splunk-sh02, address is invalid

How we can force cluster to (fully) forget old members?

0 Karma

inventsekar
Ultra Champion

ok I am not sure of this issue, but a similar one I have faced - on my cluster I had a search head, i had to make it as a peer node for a testing. It worked fine, but on master node indexer cluster gui, it was still showing as search head as down.

during my testing I had to restart splunkd on the master node. Once restart done, the old search head down issue was fixed automatically.

0 Karma

m_efremov
Explorer

I had to restart master too and there is no old peers in master's list, but they are still in logs. Splunk support specialists recommend me this solution: https://answers.splunk.com/answers/513239/remove-reference-to-host-in-mongodb.html. I will try and report here about the result.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...