Indexer Discovery Error (IndexerDiscoveryHeartbeatThread)

[amitjaywantsplu]

Hi,

I have Splunk 8.0.0 on AWS with a clustered indexer set up (1 Master and 4 indexers) and I have deployed custom test apps (with basic monitoring for windows/Linux logs) on the servers that have the forwarders installed. I have enabled the indexer discovery feature in the outputs.conf file (local folder) for these apps and on the server.conf file of the cluster master (etc/system/local) but I see the following error in the forwarder logs:

04-05-2020 16:57:53.752 +1000 ERROR IndexerDiscoveryHeartbeatThread - Error in Indexer Discovery communication. Verify that the pass4SymmKey set under [indexer_discovery:target1] in 'outputs.conf' matches the same setting under [indexer_discovery] in 'server.conf' on the Cluster Master. [uri=https://clustermaster:8089/services/indexer_discovery http_code=502 http_response="Error connecting: Connect Timeout"]

I have ensured that the pass4SymmKey attribute is the same for the outputs.conf on the forwarders and the server.conf on the cluster master (in their respective indexer discovery sections), but yet I see this error.

Any pointers that would help me resolve this?

[codebuilder]

The situation you describe generally happens when you configure a forwarder for indexer discovery but provide the hashed pass4SymmKey value from the master, rather that the plain text key.

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

[eblair84]

@codebuilder 

I'm (very) new to Splunk. How does one do this:

Update the pass4SymmKey in outputs.conf by adding the non-hashed, plain text key, then cycle the forwarder daemon.

Where do I get the non-hashed, plain text key? Also is the forwarder daemon just "splunk" on the forwarder machine?

Thanks,

Chris