Why does "splunk show shcluster-status" show "last_conf_replication: Pending" after upgrade from 6.3.2 to 6.5.0?

[daniel_kwok72]

Hi

Recently I upgraded my Splunk environment from 6.3.2 to 6.5.0. The environment has a search head cluster, an indexer cluster, and the manager server. After the upgrade, when I run the splunk show shcluster-status, I noticed there is a new status called last_conf_replication and it always shows Pending for the search head members, but the search head captain doesn't have this. The replication is running fine in the Indexer cluster, so I am not quite sure what this new option is about and how to get the replication finished. Can anyone shed some lights on this?

Thank you.

[daniel_kwok72]

Hi markusspitzli2

Did Splunk Support advise in which release this bug will be fixed? Splunk 6.6 is recently released but when I went through the release note, I couldn't find the bug fix mentioned.

[markusspitzli2]

Hi Daniel.

No. The support is still talking to the development team.

[princemanto2580]

Even for me also, other 2 Search Head Cluster member showing the same status.

last_conf_replication : Pending

Any quick suggestion to fixed it.

[markusspitzli2]

We had the same state in our 8 member Searchhead cluster with Splunk 6.5.3. After consulting the Splunk Support it turned out, that we had to use the option "mgmt_uri" instead of "server_list".
After switching the options the pending state disappeared and the cluster synched again.

This behaviour is a confirmed bug in 6.5.3. You can use either mgmt_uri or server_list according to the documentation.

you also can use both options, but you will receive this warning message:
"“It is not advised to use servers_list and mgmt_uri at same time”"

[gburgett_splunk]

The docs describe the new last_conf_replication state field in the shcluster-status command:

"The last_conf_replication field indicates when the member last pulled a set of configurations from the captain"

http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/ViewSHCstatus#Show_cluster_status

So, it makes sense that the master will not have this status field. Not sure why all of the members would be stuck in a "Pending" state after an upgrade though. Maybe there is just a lot of config information to pull down? Are you seeing any other SHC related log messages that might indicate a problem?