Any reason why I keep getting this message below and how to resolve it?

[afolabia]

Search peer XXXX(My Indexer) has the following message: Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=MMMMMM(my cluster) rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=XX..XX.XX.XX mgmtport=8888 (reason: http client error=Winsock error 10054, while trying to reach XX.XX.XX.XX. Make sure pass4SymmKey is matching if the peer is running well. [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=12344444 add_type=Clear-Masks-And-ReAdd base_generation_id=689 batch_serialno=1 batch_size=4 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=123444 mgmt_port=8888 name=43A9B5B9-076B-439C-9AEA-3E41708C75D4 register_forwarder_address= register_replication_address= register_search_address= replication_port=9200 replication_use_ssl=0 replications= server_name=XXXXXXXX site=default splunk_version=7.0.2 splunkd_build_number=03bbabbd5c0f status=Up } ].

‎12‎/‎6‎/‎2018‎ ‎6‎:‎58‎:‎03‎ ‎PM
✗

[DavidHourani]

Hi @Afolabia,

Could you check if all your Splunk management ports are open on your firewall ? You want to be sure that the peers communication with the CM is uninterrupted, possibly some changes on the network could be causing that.

Also check if you can see all your indexers in "indexer clustering" . If you can't then please go into server.conf on the indexers you don't see and re-enter your pass4SymmKey. Encrypted pass4SymmKey on your servers shouldn't have the same value as each uses a different splunk.secret as a salt for the encryption.

Cheers.
David

[whrg]

The error message says "Make sure pass4SymmKey is matching if the peer is running well."
Perhaps the pass4SymmKey does not match on all servers in your cluster?

[afolabia]

pass4SymmKey does matched. The error goes away on it's own and comes back every other week.