I setup a search head cluster on 3 search heads:
[root@deploy-searchhead01 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead01.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!
[root@deploy-searchhead02 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead02.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!
[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk init shcluster-config -auth admin:changeme -mgmt_uri https://deploy-searchhead03.bigdata.emc.local:8089 -replication_port 34567 -secret Password01!
I bootstrapped searchhead03 as the captain with the 3 search head:
[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list "https://deploy-searchhead01.bigdata.emc.local:8089,https://deploy-searchhead02.bigdata.emc.local:808..." -auth admin:changeme
When I run the command "splunk show shcluster-status", only the captain is shown in the Members:
[root@deploy-searchhead03 ~]# /opt/splunk/bin/splunk show shcluster-status -auth admin:changeme
Captain:
dynamic_captain : 1
elected_captain : Thu Oct 20 17:35:52 2016
id : 54B57E56-AB72-4569-AE07-A5B8049C5690
initialized_flag : 0
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0
Members:
deploy-searchhead03.bigdata.emc.local
label : deploy-searchhead03.bigdata.emc.local
mgmt_uri : https://deploy-searchhead03.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.78:8089
status : Up
The output is same on all the 3 search heads. Please help and let me know what's wrong. The Splunk version is 6.5.0.
Thank you.
issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log
issue fixed. Check the log for reason: splunk/var/log/splunk/splunkd.log
Check the log for reason: splunk/var/log/splunk/splunkd.log
It will show the detial event why the other members are not added into the cluster.
In my case, as I cloned the Splunk search head VM, all the search heads' guids are same. That's why they cannot be added into one cluster.
Resolution:
rm /opt/splunk/etc/instance.cfg
splunk restart
Glad you got it resolved! But could you share what you found and how you fixed the issue? Somebody may stumble across this post in the future, and maybe your experience will be able to help them.
I configure the server.conf and add the pass4SymmKey = changeme under [shclustering] and restart the splunkd.
I rebuild the SH cluster using the -secret changeme and -shcluster_label shcluster1, however, the output is same, no other members shown in the member list:
Captain:
dynamic_captain : 1
elected_captain : Fri Oct 21 12:20:43 2016
id : 6F21DCB3-0E06-4AA0-8E1C-7FE2D2712588
initialized_flag : 0
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0
Members:
deploy-searchhead01.bigdata.emc.local
label : deploy-searchhead01.bigdata.emc.local
mgmt_uri : https://deploy-searchhead01.bigdata.emc.local:8089
mgmt_uri_alias : https://172.16.1.72:8089
status : Up
have you looked in the /opt/splunk/var/log/splunkd.log on the captain or members to see if there are any errors/warnings in there around running those commands or communication between servers?
Is SSL enabled by default for the management port? No idea, but was wondering if http/https was in play here at all.
As I used the different password for -secret with the instance password "changeme"
I can see in the server.conf file, the pass4SymmKey = changeme is under [general]. However, under [shclustering] there is no pass4SymmKey configuration..