Why do I keep getting prompted for a username and password trying to add a member to a search head cluster?

[rmorlen]

I am trying to add a new member to a search head cluster.

Init the server with:

$SPLUNK_HOME/bin/splunk init shcluster-config -mgmt_uri https://server1:8089 -replication_port 8095 -replication_factor 3 -conf_deploy_fetch_url https://deployer:8088 -secret secretpw

Trying to add the server:

splunk add shcluster-member -current_member_uri https://server2:8089

Get prompted for username and password. Use admin and pw, but keeps prompting for username and password. Also tried a pw of changeme but then I get authentication failed.

I currently have 3 searchheads in the cluster.

[arahut_splunk]

The reason this is happening is because the pass4SymmKey/shared-secret on your new member was not matching
the secret of the rest of the cluster.

When you ran the CLI command splunk add shcluster-member -current_member_uri https://server2:8089,
an authentication token was generated on the search head server1 and put in its $HOME/.splunk directory.

add shcluster-member proxies the request to one of the current members (server 2) and tries to get the current
node added via the captain. This proxy process fails due to authentication, when the invalid token is used.

What you will need to do is.

 1. bring down the node that is being added and fails.
 2. go to server.conf of that node and change its pass4SymmKey to correct shared secret.   or re  init the node.
 3. delete the $HOME/.splunk folder so that the CLI gets a new key
 4. restart splunk
 5. try the add shcluster-member command again.

[rmorlen · Accepted solution]

I completely stopped all searchheads and reinitialized the cluster. Brought everything back up and was able to get it working.

[esix_splunk]

You should check in splunkd and see what errors are being given. Most likely your secret password for the cluster is incorrect.

Also, is your deployer on 8088?

[esix_splunk]

What is the message in splunkd.log say?

[rmorlen]

Yes deployer is 8088.

Reset passwords and am still seeing the same issues.