Splunk Search

Why "show shcluster-status" only shows the captain in Members list?

guotao4321
Path Finder

I setup a search head cluster on 3 search heads:

[root@deploy-searchhead01 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead01.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead02 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead02.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead03.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!

I bootstrapped searchhead03 as the captain with the 3 search head:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://deploy-searchhead01.bigdata.emc.local:8089,https://deploy-searchhead02.bigdata.emc.local:808..." -auth admin:changeme

When I run the command "splunk show shcluster-status", only the captain is shown in the Members:

[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk show shcluster-status -auth admin:changeme

Captain:
dynamic_captain : 1
elected_captain : Thu Oct 20 17:35:52 2016
id : 54B57E56-AB72-4569-AE07-A5B8049C5690
initialized_flag : 0
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead03.bigdata.emc.local
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.78:8089
status : Up

The output is same on all the 3 search heads. Please help and let me know what's wrong. The Splunk version is 6.5.0.
Thank you.

Tags (1)
1 Solution

guotao4321
Path Finder

issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log

View solution in original post

guotao4321
Path Finder

issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log

guotao4321
Path Finder

Check the log for reason: splunk/var/log/splunk/splunkd.log
It will show the detial event why the other members are not added into the cluster.

In my case, as I cloned the Splunk search head VM, all the search heads' guids are same. That's why they cannot be added into one cluster.

Resolution:
rm /opt/splunk/etc/instance.cfg
splunk restart

0 Karma

maciep
Champion

Glad you got it resolved! But could you share what you found and how you fixed the issue? Somebody may stumble across this post in the future, and maybe your experience will be able to help them.

0 Karma

guotao4321
Path Finder

I configure the server.conf and add the pass4SymmKey = changeme under [shclustering] and restart the splunkd.

I rebuild the SH cluster using the -secret changeme and -shcluster_label shcluster1, however, the output is same, no other members shown in the member list:

Captain:
dynamic_captain : 1
elected_captain : Fri Oct 21 12:20:43 2016
id : 6F21DCB3-0E06-4AA0-8E1C-7FE2D2712588
initialized_flag : 0
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

Members:
deploy-searchhead01.bigdata.emc.local
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.72:8089
status : Up

0 Karma

maciep
Champion

have you looked in the /opt/splunk/var/log/splunkd.log on the captain or members to see if there are any errors/warnings in there around running those commands or communication between servers?

Is SSL enabled by default for the management port? No idea, but was wondering if http/https was in play here at all.

0 Karma

guotao4321
Path Finder

As I used the different password for -secret with the instance password "changeme"
I can see in the server.conf file, the pass4SymmKey = changeme is under [general]. However, under [shclustering] there is no pass4SymmKey configuration..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...