Deployment Server unable to phone home

[francis_larkin]

I recently upgraded from 4.3 to 6.3.3 using an RPM update. I followed the recommendations of Splunk and upgraded from 4.3 to 6.0 and then to 6.3.3. All went well. I now see on my deployment server that it cannot phone home.

I see the error...
04-22-2016 12:50:10.302 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.28.yyy.yy_8089_mydeployment_server
04-22-2016 12:50:14.746 +0000 INFO ClientSessionsManager:Listener_AppEvents - Received count=3 AppEvents from DC ip=172.20.xxx.xxx name=defaultforwarder
04-22-2016 12:50:14.746 +0000 INFO ClientSessionsManager - ip=172.20.xxx.xxx name=defaultforwarder Updating record for sc=jbossStatistics app=JBossStatistics: action=Install result=Ok checksum=9537295459324277384
04-22-2016 12:50:14.746 +0000 INFO ClientSessionsManager - ip=172.20.xxx.xxx name=defaultforwarder Updating record for sc=jbossUF app=JBOSS_UF: action=Install result=Ok checksum=9364516596849177361
04-22-2016 12:50:14.746 +0000 INFO ClientSessionsManager - ip=172.20.xxx.xxx name=defaultforwarder Updating record for sc=systemCheck app=unix: action=Install result=Ok checksum=14281476786925036628
04-22-2016 12:50:15.549 +0000 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin

Actually I am not sure if this message is from the deployment server or one of the other but I do know...
1) All my indexers (16) are working.
2) All my 700+ forwarders are forwarding to the indexers.
3) All the indexers and forwarders are able to phone home because I see them come into the Deployment server.
4) We are able to query and get no error messages complaining about missing indexers.
5) The data looks good from all sources.

I just cannot figure out why the Deployment server is not phoning home.

It apparently is trying to use the login and password. I assumed all the old passwords I was using were preserved.

FYI: Some name and IP were masked to protect the innocent.

Any help is appreciated.

[andykuhn]

The Deployment Server represents 'home' to ONLY those nodes running a forwarder client (who can then 'phone home' to the deployment server). The deployment server should not be running a universal or heavy forwarder because you are running the core enterprise software, wherein, you are directing output (_internal index goodies) directly to the indexer (forwarding and indexing configuration). Both the forwarder and the core installation utilize the splunkd process, so cannot physically both be running on the same node. The deployment server is 'home' and the source of your forwarder inputs sent to your forwarding nodes. I hope this helps.

In Splunk search for the log entry above and take note of the 'source' and 'host' values. This will tell you which node is responsible for generating the event.