I read a lot of comments about the error DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
To summarize what I found out:
This error is about the connection between DC and DS, so it doesn't affect the possibility of your forwarder to upload the logs. But if there's a problem with DC - DS communication your DC can be unable to deploy apps to your forwarders (like outputs or inputs ), so your forwarder doesn't know where to take logs and where to forward.
What you can check on forwarder:
Is your DS uri set properly: $SPLUNK_HOME/bin/splunk show deploy-poll
Is there a network connection between DC and DS. For example using telnet <DS-uri> 8089 (if you didn't change management port)
Is your forwarder showing on DS in clients tab and when it phoned home for the last time.
Did your forwarder get the apps from DS. They shoud be located in directories like this: $SPLUNK_HOME/etc/apps/<your_app_name>/local/
The version of Splunk forwarder.
In my case my forwarder wasn't showing on DS in clients tab, but the DS uri was correct. And there were no problems with the network connectivity. But I resolved the issue by upgrading the forwarder from 6.2.5 to 7.0.2. And on indexers I have 7.2.1.
Hope it's helpful.
... View more