Getting this pop-up message in splunk console Failed to synchronize configuration with KVStore cluster?

[Hemnaath]

Hi All, Currently I am facing an issue in one of the cluster search head member and i am getting this pop-up message in the splunk forwarder management console.

Message details:
Search peer host01.xxxx.com has the following message: Failed to synchronize configuration with KVStore cluster. replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the "force" argument to override.

Splunkd.log details:

11/9/17
5:40:22.128 AM

11-09-2017 05:40:22.128 -0500 ERROR KVStoreBulletinBoardManager - Failed to synchronize configuration with KVStore cluster. replSetReconfig should only be run on PRIMARY, but my state is SECONDARY; use the "force" argument to override

The above message started after pushing Splunk Add-on F5 LTM to the search head cluster member from the Deployer instances.
It restarted all the search head cluster members. And we started getting the above pop-up for one of the cluster member.

Kindly guide me how to fix this issue.

thanks in advances.

[Hemnaath]

Hi Kunalmao, Good Evening, thanks for your effort on this, currently this is the permission level set for the splunk.key.

$SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
-rw------- 1 splunk splunk 88 May 16 09:49 splunk.key

Hey I have gone through the link provide by you as per that we have only 3 cluster member in our environment. So we are in the odd number, I hope there is no need to remove the cluster member.

And I have another question.

1) DO we need to execute the below command on all the search head member to clear the content inside the kvstore/mongo. Or is it good to execute only in the affected search head member.
./splunk clean kvstore --local

2) Will there be any impact if we are going to clean/removing the content present in the kvstore/mongo. As this is the first time, I am coming across this type of issue.

Kindly guide me on this.
thanks in advance.

[kunalmao]

What is the size of your search head cluster ?

Mostly it is the case of permission $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key , change the permission of this file

chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

If still the error persists and you are running search head cluster then please refer my answer below

https://answers.splunk.com/answers/550274/kv-store-failing-at-shc.html#answer-588554