Upgraded Splunk Universal Forwarder 6.3.1 (build f3e41e4b37b2) from 5.07 and I was not getting data to indexers and Apps were not updating

[dmacgillivray]

Hello Splunkers,

I had an issue with a Splunk Universal Forwarder 6.3.1 (build f3e41e4b37b2) upgrade recently
and I wanted to share with you the issue and its resolution. Knowing that this could happen to anyone.

After the upgrade to 6.3.1 from version 5.07 this particular host would not send any data.
I must have checked the same serverclass entry 50 times. Then the app itself another 50 times as
that would show old timestamps as well under C:\Program Files\SplunkUniversalForwarder\etc\apps

After working with Support we found the below errors and sucesses in
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log on the forwarder.

Errors:
12-16-2015 10:18:54.142 -0400 WARN TcpOutputFd - Connect to xx.xxx.xx.xxx:9997 failed. No connection could be made because the target machine actively refused it.
12-16-2015 10:18:54.142 -0400 ERROR TcpOutputFd - Connection to host=xx.xxx.xx.xxx:9997 failed
12-16-2015 10:18:54.142 -0400 WARN TcpOutputProc - Applying quarantine to ip=xx.xxx.xx.xxx port=9997 _numberOfFailures=11
12-16-2015 13:43:33.377 -0500 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Success messages (outputs.conf was fine 🙂 but nothing getting there or so I had thought.
12-16-2015 15:20:36.812 -0500 INFO TcpOutputProc - Connected to idx=xx.xxx.xx.xxx:9997 (indexer 1)
12-16-2015 15:21:06.920 -0500 INFO TcpOutputProc - Connected to idx=xx.xxx.xx.xxx:9997 (indexer 2)

[dmacgillivray · Accepted solution]

Solution:
As it turns out this windows forwarder was renamed and provided a new IP at some point.
After the rename the below files were never updated as I assumed one can just upgrade.

Understanding Splunk Heirarchy, I should have known that these files will not change unless manually updated because they
are in a "local" directory.

C:\Program Files\SplunkUniversalForwarder\etc\system\local\inputs.conf
C:\Program Files\SplunkUniversalForwarder\etc\system\local\server.conf

When I found this issue I changed the server name in the above files and restarted the forwarder which then began to send data.