Deployment Architecture

In the Monitoring Console, why does Search Head Clustering dashboard display "no captain available"?

jmccallhbo
Explorer

In the Monitoring Console on 6.5, viewing the Search Head Clustering: Status and Configuration dashboard my SHC members are listed as up and have a consistent baseline but in the 'number of unpublished changes' column it says 'no captain is available'. Is this an error state? Can my SHC members not send changes to the SHC captain?

I had recently pushed a change from the deployer and the SHC members has an inconsistent baseline. So I then did a resync to fix the problem. Being that the resync is destructive I would like to not have to do that after every time I push configuration changes from the deployer.

0 Karma

jmccallhbo
Explorer

I've resolved the issue I was having. When the monitoring console shows no captain available under unpublished changes it is indeed an error and means that the peers can not replicate changes to the captain. This can also be seen by running show shcluster-status and noticing that for the peers the last_conf_replication is in a state pending. Apparently the peers can still join the cluster even though they are unable to replicate changes with the master.

It turns out the problem I was having was networking related. Originally I had instances with only a private network interface. I later had to add a public interface to all my search heads and I believe this caused the problem because my firewall was only allowing traffic from the private ip addresses and I assume the default behavior of the search head cluster members was to replicate using the public interface. The solution I found was to set register_replication_address = <IP> in server.conf on the search heads which identifies the address used for receiving replication data.

aaraneta_splunk
Splunk Employee
Splunk Employee

Hi @jmcallhbo - Glad to hear that you found the solution to your question. Please don't forget to click "Accept" below your answer so that other users can easily find it, especially if they're having the same issue. Thank you!

0 Karma

jmccallhbo
Explorer

Unfortunately I have not resolved the issue. The members of my cluster have begun to show "no captain available again" and I seem to be unable to recover. I think this may be related... Also, the version I'm using is 6.5.1.

RaftSimpleFileStorage - Error reading metadata1: error in raft mdata file for node version

0 Karma

aaraneta_splunk
Splunk Employee
Splunk Employee

No problem. We'll leave this question open to see if any other users want to try answering.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...