I am setting up Distributed Search for the first time. When I enter the Peer and Authentication information, I get the red error msg;
Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer https://splunk11:8089: Connect timed out.
Splunk is running on the peer, but when I try to manually telnet to 8089, the connection is immediately refused. When I do a netstat -an on the box, it is not showing anything listening on 8089.
Troubleshooting Steps:
1.Login to Splunk indexer
& Splunk search head
(ssh)
2.From indexer : Check splunk service is running : $SPLUNK_HOME/bin/splunk status
Possible Error:
splunkd xxxx was not running.
Stopping splunk helpers...
couldn't send SIGTERM to pid xxxx: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.
From indexer : Goto >> $SPLUNK_HOME/var/run/splunk/
rm splunkd.pid
$SPLUNK_HOME/bin/splunk status
$SPLUNK_HOME/bin/splunk start
It should work.
OR
Possibly Firewall issues:
From indexer :
telnet localhost 8089
Check 8089 port is listening
From search head:
telnet splunk-indexer 8089
check u can able to reach 8089 port of indexer
You should have allowed in firewall rules.
I noticed on mine that the new search head was listening on 8089 but the older indexer was listening on 8090 (just FYI)
remote peer not listening on 8089? sounds wired, restart it and check if splunkd 8089 is opened
Splunk was configured to listen on a different/non-standard port. Once I solved that, I was able to get the Instances talking to each other.