I have message popping on my UI messages " Search head cluster member (sh3) is having problems pushing configurations to the search head cluster captain (sh4). Changes on this member are not replicating to other members. We have 4 search heads with dynamic captain
I have ran the ./splunk resync shcluster-replicated-config command and also ran the /splunk clean kvstore --local . Still it is showing the error
Any other workaround for this
Can you try reducing the value of conf_replication_max_push_count and give it some time to catch up with the syncing in small pushes only.
At below post, its been resolved this way -
https://answers.splunk.com/answers/539335/why-is-a-search-head-cluster-member-not-replicatin.html
Please accept as answer if it resolves your case.
Thanks
Where can I change the conf_replication_max_push_count ,should I do It on the captain and push the changes?
you can set it at server.conf. A direct manual update to the .conf file would not be replicated but however through Web OR CLI OR REST they should be.
Check this for SHC replication -
https://docs.splunk.com/Documentation/Splunk/7.3.0/DistSearch/HowconfrepoworksinSHC
I see the following message in the logs
ERROR ConfReplicationThread - Error pushing configurations to captain=https://hesplsrhc003com:8089, consecutiveErrors=173 msg="Error in acceptPush: Non-200 status_code=400: ConfReplicationException: Cannot accept push with outdated_baseline_op_id=d211555c171b47ccc8c7fe5f5588856096680c0e; current_baseline_op_id=c6235cd928e2196f63bb3cb292a55a2687467c46": Search head clustering: Search head cluster member (https://hesplsrhc001.com:8089) is having problems pushing configurations to the search head cluster captain (https://hesplsrhc003.com:8089). Changes on this member are not replicating to other members.
@vrmandadi Check the splunkd logs for the resync if it happened correctly or there was some error.
Also this post although is for captain but may help - https://answers.splunk.com/answers/525189/how-to-resolve-error-error-pulling-configurations.html