Splunk client not registering with deploy server

[jkfierro]

Splunk 7.0.x

We have VM template. When we deploy VMs, the hostname changes for the machine, and I can update the inputs.conf and server.conf files to reflect the change. It all works fine, and clients report into Splunk indexer as a deployment client.

For several hosts, we had to change hostname again. I updated these same conf files again to reflect the hostname change. However they no longer appear in indexer as deployment client. They are forwarding data though, using older forwarder configuration files.

I’ve tried deleting and resetting config files in the forwarder. I’ve tried uninstall and reinstall of the forwarder. I’ve tried re-adding deploy client via command line for the forwarder. I’ve tried searching the indexer for duplicate clients so I could issue a reset to the client.

Still, it does not register anymore with deploy server. How do I get it to register as a client to the deploy server again?

Any other ideas?

[jkfierro · Accepted solution]

I believe this is the resolution:

https://answers.splunk.com/answers/542872/what-do-i-look-at-in-splunkdlog-to-troubleshoot-de.html

[jkfierro]

I see this in the Deploy Server logs for multiple hosts:

05-04-2018 07:39:45.471 -0400 WARN ClientSessionsManager - Client with Id '' has changed some of its properties on the latest phone home.Old properties are: ip= dns= hostname= build=c8a78efdd40f uts=linux-x86_64 name=. New properties are: ip= dns= hostname= build=c8a78efdd40f uts=linux-x86_64 name=.

[jkfierro]

1 - checks in fine
2 - i believe we do use ssl, though we don't mess with any of the ssl settings
3 - i can telnet from src to dst over 8089
4 - i can register apps, if it would show up as a client

So, I'm noticing, these boxes that had their hostname changed, periodically they appear very briefly in the Client list on the Splunk Indexer / Deploy Server. Then they disappear just as quickly.

[woodcock]

Here are the possible problems and the solutions:

1: Forwarder is pointed to the wrong place; check with this command

$SPLUNK_HOME/bin/splunk btool deploymentclient list --debug

2: Forwarder is using wrong/misconfigured/unsupported SSL cert/TLS; you are on your own here and there are MANY bugs and version variants. It is a deep topic but probably you are NOT doing SSL for DC/DS (most people are not).

3: Forwarder is blocked by firewall

| tstats summariesonly=t count
FROM datamodel=Network_Traffic
WHERE index=*
AND (All_Traffic.dest_port = 8089)
AND (All_Traffic.action = "blocked" OR All_Traffic.action = "dropped")
BY All_Traffic.src All_Traffic.dest

4: Forwarder is not whitelisted for the appropriate apps on the DS. You have to know this and verify it manually.