Getting Data In

Handshaking issue with deployment server

rameshlpatel
Communicator

Hi,

I am getting following error message on universal forwarder logs:

11-10-2013 17:43:38.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-10-2013 17:43:46.141 +0530 ERROR HTTPClient - Should have gotten at least 3 tokens in status line, while getting response code. Only got 0.
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Secure HTTP POST failed: Unknown read error
11-10-2013 17:43:46.141 +0530 INFO HttpPubSubConnection - Could not obtain connection, will retry after=83 seconds.
11-10-2013 17:43:50.750 +0530 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

0 Karma

ebaileytu
Communicator

That is really interesting since I am running the UF on the same host as the search head but I also have the management port for the UF turned off

[httpServer]
disableDefaultPort = true

and it is talking to another server for its deployment information.

I am going to turn the UF off for a bit to see if it helps.

Thanks!

0 Karma

ebaileytu
Communicator

another common error message from the captain for each member that stops running scheduled searches

05-27-2017 15:02:32.502 -0500 ERROR SHCMasterArtifactHandler - failed on handle async replicate request sid=scheduler_adminxxxxRMD52a88c92ed83e8b0e_at_1495915200_11776_2E1C054F-9A8B-4D4A-BBC0-29F0562C7AED err='targetPeer="member", targetGuid="88275523-AE18-4CD9-AD67-7956E06449C1" cannot be valid target for artifactId=scheduleradmincaptain_RMD52a88c92ed83e8b0e_at_1495915200_11776_2E1C054F-9A8B-4D4A-BBC0-29F0562C7AED srcPeer="captain", srcGuid="A6A2F1D5-37C9-419C-A85E-A42376EDD483" reason="peer already has artifact"'

0 Karma

bmcclary_splunk
Splunk Employee
Splunk Employee

This answer is unlikely to help in most cases, however, I was getting this error on my local laptop (lab) where I had Splunk Enterprise (Deployment server) and Splunk Universal Forwarder (UF) running with the UF's targetUri setting in deploymentclient.conf pointing to localhost (local machine's IP actually). The issue of course was they were both using 8089 for mgmt port. By changing the port on my Enterprise instance to 8091 and restarting the enterprise instance running the deployment server, issue was resolved. Use
./splunk set splunkd-port 8091 on my DS
Restart DS instance

0 Karma

bohanlon_splunk
Splunk Employee
Splunk Employee
0 Karma

neelamsantosh
Path Finder

I resolved the issue by flushing iptables,hope it can resolve your issue too.

iptables -F

cheers

ohulea
Engager

Hi I can confirm that iptables -F works in a homelab.

0 Karma

ebailey
Communicator

We had exactly the same issue with the same error message and we struggled to figure it out - this turns out to be a MTU setting issue with a data center switch. Makes sense, given the ability to telnet to a port, but the web service then fails to work.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Check also make sure the local firewall / iptables is permitting TCP8089 to the DS host, and since these are different zones also confirm that the actual clients can connect to 8089 and not just your machine.

After that, make sure there is twoway (inbound / outbound) traffic through the firewall / acl for 8089 to the DS enabled.

WedbushITOps
Engager

make sure the port (8089 in my case) is open

Conradj
Path Finder

I'm getting the error too.

But only for deployment clients in a particular network zone.

I can telnet to the deployment server on TCP 8089 fine, but the clients get the errors above.

At this stage I think it is a routing issue, our firewall team has been involved but have not detected any drops.

C.

0 Karma

lukejadamec
Super Champion

What do you have in your deploymentclient.conf on the forwarder?

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...