Splunk Search

Search Head Clustering fails to work..

dilipbailwal
Path Finder

As per the documents, i have successfully configured deployer and then further initialized all the Search Members too.
Now, while Bringing up Captain continuously failed .. Not sure why we have to define captain if the Election is Dynamic.

Anyways, here is the error what i am getting after executing following command,
./splunk show shcluster-status -auth admin username:password

> This node is not the captain of the
> search head pool, and we could not
> determine the current captain. The
> pool is either in the process of
> electing a new captain, or this member
> hasn't joined the pool.

after a while i checked and still captain wasn't elected.
I also ran bootstrap command to bring up captain and it failed. with error,

> In handler
> 'shclustermemberconsensus': Failed to
> bootstrap this node as a captain.

and then one of the node where bootstrap produces result,

> In handler 'shclustermemberconsensus':
> NOT_LEADER CURRENT_STATE = FOLLOWER

0 Karma
1 Solution

dilipbailwal
Path Finder

Issue is resolved.

The problem was, one of the servers in SH cluster had a wrong mgmt_uri.. hence it wasn't connecting..
After correcting the server, it worked and also i did bootstrap on another server as it has to be executed only once on a particular server.

Thanks amiracle for your reply

View solution in original post

dilipbailwal
Path Finder

Issue is resolved.

The problem was, one of the servers in SH cluster had a wrong mgmt_uri.. hence it wasn't connecting..
After correcting the server, it worked and also i did bootstrap on another server as it has to be executed only once on a particular server.

Thanks amiracle for your reply

amiracle
Splunk Employee
Splunk Employee

Did you try modifying the server.conf and make sure that your mgmt_uri is the same as the search head that you are trying to start?

Here is a sample entry in my server.conf (assuming my search head cluster member is splunk05 and my deployer is splunk01):

[general]
pass4SymmKey = <some_passkey>
serverName = splunk05

[replication_port://34567]

[shclustering]
conf_deploy_fetch_url = https://splunk01:8089
disabled = 0
mgmt_uri = https://splunk05:8089
pass4SymmKey = <some_encrypted_passkey>
replication_factor = 3
id = <some_generated_id>

I had the same problem and it turns out my mgmt_uri was pointing to the previous captain, something I think I did by accident during my adding of the cluster member.

hitesh_kanchan
Explorer

Did you try running "./splunk bootstrap" in another Search head member as well.

If that doesn't help, try setting pass4Symmkey=someValue in [shclustering] in server.conf and perform "./splunk init with -secret someValue" in the Search head members.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...