Splunk Search

Tried to add a search peer: Error while sending public key to search peer: Connection closed by peer

Jarohnimo
Builder

(attempting 1 Indexer, +1 SH setup)

Tried the Following the Instructions from Splunk
1. Log into Splunk Web on the search head and click Settings at the top of the page.

  1. Click Distributed search in the Distributed Environment area.

  2. Click Search peers.

  3. On the Search peers page, select New.

  4. Specify the search peer, along with any authentication settings.

Note: You must precede the search peer's host name or IP address with the URI scheme, either "http" or "https".

  1. Click Save.

    1. Repeat for each of the search head's search peers.

I've confirmed that my management port is open. Was able to confirm that all my forwarders are also connecting via 8089 so i'm not sure why it's saying connection closed ... The search peer has everything on it.

It produces: Encountered the following error while trying to save: In handler 'distsearch-peer': Error while sending public key to search peer: Connection closed by peer

I've also tried to add make this search head a slave to my indexer that is the license master but when I add http://myserver.domain.com:8089 I get similar results: Bad Request — In handler 'localslave': editTracker failed, reason='Unable to connect to license master: http://myserver.domain.com:8089 Unknown read error'

This is wild because I'm able to telnet fine, when I netstat it shows connection to 8089 on the search peer, fire wall is wide open but still no connection error... Is there any steps I have to do prior? Does this only work with https? Are there any steps other than changing the default password on both nodes that I have to do? Do I need to adjust anything in the DMC prior? or perhaps in any of the config files to prep? Thanks for your help

0 Karma
1 Solution

Jarohnimo
Builder

I Figured it out, Both messages were related to me just not knowing what the heck i'm doing.

Licensing issues: The account that i was logged in with was an Admin in Splunk on both nodes but didn't have any rights on the server. I then tried switching the Search HEad to Slave with my Server Admin account (Also splunk admin) and then it worked. I found one of my issues was on the new server (SH) i jumped the gun adding the license to this box and then splunk got mad. You only need it on the master.

Adding the Distributive search answer:

I was typing http://myserver.domain.com:8089 (Even though its an option that suppose to work) for me it didn't. Just go with their first option in: Servername:Port# it will work. (Remember to use an account that has local admin rights on both boxes as well is a splunk admin... preferably a domain account)

View solution in original post

neelamsantosh
Path Finder

Check the host level firewall. as often we avoid it in our troubleshooting.
enable/permit the port in firewall using:
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent

and reload the config list:
firewall-cmd --list-all

Now configure the search peers.

Jarohnimo
Builder

I Figured it out, Both messages were related to me just not knowing what the heck i'm doing.

Licensing issues: The account that i was logged in with was an Admin in Splunk on both nodes but didn't have any rights on the server. I then tried switching the Search HEad to Slave with my Server Admin account (Also splunk admin) and then it worked. I found one of my issues was on the new server (SH) i jumped the gun adding the license to this box and then splunk got mad. You only need it on the master.

Adding the Distributive search answer:

I was typing http://myserver.domain.com:8089 (Even though its an option that suppose to work) for me it didn't. Just go with their first option in: Servername:Port# it will work. (Remember to use an account that has local admin rights on both boxes as well is a splunk admin... preferably a domain account)

anand_singh17
Path Finder

Sometimes the service is not reachable, so gives this error.

You can use inbuilt user account or domain account.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...