Getting Data In

Splunk Intermitten Indexer Cluster issues after upgrade to 7.2.1

johnward4
Communicator

05-17-2019 00:35:38.768 -0700 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=clustermaster:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=171.67.54.9 mgmtport=8089 (reason: non-zero pending job count=10, guid=39064DB5-4A78-4111-AB9A-DA5A8B7A886B). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 add_type=ReAdd-As-Is base_generation_id=9900 batch_serialno=1 batch_size=154 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=10519 latest_bundle_id=79DE6CD58B53E34AED36B939B94F8E23 mgmt_port=8089 name=39064DB5-4A78-4111-AB9A-DA5A8B7A886B register_forwarder_address= register_replication_address= register_search_address= replication_port=8091 replication_use_ssl=0 replications=client_som_irt_unix~480~B040E664-4DDA-4073-8FD5-4A68619C94C3::39064DB5-4A78-4111-AB9A-DA5A8B7A886B::D9480470-6D8A-4D8B-B6A6-5EF0F926676D server_name=splunkidx04.domain.com site=site1 splunk_version=7.2.1 splunkd_build_number=be11b2c46e23 status=Up } ].

Has anyone experienced a similar issue after the upgrading from Splunk Enterprise 6.5.4 to 7.2.1 in a index clustered environment?

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Yes, you most likely have one or more pass4SymmKey values set incorrectly.

Ensure these values are set correctly within /opt/splunk/etc/system/local/server.conf

On the master:

[general]
serverName = fqdn_of_your_master
pass4SymmKey = cluster_wide_password_shared_across_all_nodes

[clustering]
mode = master
replication_factor = int_value
search_factor = int_value
pass4SymmKey = password_unique_to_this_index_cluster # Note, this is NOT the same p/w as under [general]
cluster_label = a_unique_label_for_your_index_cluster # Note, indexers only

[indexer_discovery]
pass4SymmKey = password_unique_to_indexer_discovery # assuming you use this

On each indexer:

[general]
serverName = fqdn_of_your_indexer
pass4SymmKey = cluster_wide_password_shared_across_all_nodes

[clustering]
mode = slave
pass4SymmKey = password_unique_to_this_index_cluster 
master_uri = full_uri_and_port_of_your_master # e.g. https://mymasternode.com:8089

Then cycle the master, and each indexer one by one.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...