We are building a single-site pilot environment with the following layout:
1 x Deployment and License Manager
3 x Search heads (configured in a SH cluster)
3 x Indexers
1 x Indexer Cluster Master
We have the indexers set up and sharing data and we even have a test forwarder sending data to an index on those clusters, however, when we connect to the search heads (either directly or via our load-balanced IP in front of them) the search does not see any data whether from the internal indexes or the external test index we built for our test data.
We ran the /opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list
command and listed all of our search heads. We also ran the /opt/splunk/bin/splunk edit cluster-config-mode searchhead -master_uri
on the captain.
We've read through the http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/SHCandindexerclusterdocumentation but we're not clear whether we need to run edit cluster-config -mode searchhead -master_uri
command on each of the search heads. That is what we've done, but it doesn't appear to resolve the issue.
Hi joshuabiggley, You will want to setup each index cluster slave as a search peer on each search head cluster member. I believe this will resolve your issue. Please let me know how it works for you! 😄
Hi joshuabiggley, You will want to setup each index cluster slave as a search peer on each search head cluster member. I believe this will resolve your issue. Please let me know how it works for you! 😄
We had already added the cluster slaves as search peers on each search head cluster member. We used the command below...
splunk add search-server -host : -auth : -remoteUsername -remotePassword
When we tried to re-run the command we got an error about the cluster slave already existing. After a little more digging we realized that we need to connect to each of the search heads and assign the admin role that ability to see the indexes. We had done this on all of the indexers, but had not done it on the search heads.
http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/Connectclustersearchheadstosearchpeers
You can also add search peers through Splunk Web on each search head. To do this, you must first unhide the hidden settings, as described in "The Settings menu." Then follow the instructions in "Add search peers to the search head."
We can see data on each of the search heads from the indexer. Now I just need to figure out:
1) How to replicate those role settings to all servers without having to manually touch them?
2) Why the DMC doesn't see the search heads (or license server for that matter!)?
Thanks for helping us find the correct path even if it wasn't the exact right answer.
Also enabled the DHC and confirmed that we are not seeing a search head cluster defined there either.
Here is the output of the /opt/splunk/bin/splunk list cluster-config command:
config
access_logging_for_heartbeats:1
cxn_timeout:60
disabled:0
forwarderdata_rcv_port:?
forwarderdata_use_ssl:0
heartbeat_period:0
heartbeat_timeout:60
master_uri:https://[IP of the index cluster master]:8089
max_peer_build_load:5
max_peer_rep_load:5
mode:searchhead
multisite:false
percent_peers_to_restart:10
ping_flag:1
quiet_period:60
rcv_timeout:60
rep_cxn_timeout:60
rep_max_rcv_timeout:600
rep_max_send_timeout:600
rep_rcv_timeout:60
rep_send_timeout:60
replication_factor:3
replication_use_ssl:0
restart_timeout:60
search_factor:1
search_files_retry_timeout:600
send_timeout:60
site:default