Splunkforwarder 5.0 - TcpOutputProc - Cooked connection to ip=... timed out

[Ovi]

Ok, this is really weird (... and annoying 🙂

Deployed splunkforwarder 5.0 last night on 4 identical servers at 2 sites:
cs1server1,cs1server2,
cs2server1,cs2server2

The forwarder works just fine on both xxserver1s but does not want to connect to the indexer on the other xxserver2s - throws this message in splunkd.log:

11-28-2012 11:32:56.847 -0500 WARN TcpOutputProc - Cooked connection to ip=10.152.19.16:9997 timed out

The 4 boxes are identical from all perspectives, subnets, network, hw, apps running, etc
The installation is completely automated so the exact same package was deployed
The IP and port are not blocked - telnet 10.152.19.16 9997 works just fine from all forwarders
Restarted forwarders to no avail - same message - they're never able to connect from those two servers buth they always work fine from the other two

I am out of ideas...any suggestions?

..system/local/outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk.bns:9997

[tcpout-server://splunk.bns:9997]

[Akili]

i had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path "\SplunkUniversalForwarder\ etc\ system\ local "
replace the config files under with those from:
path \SplunkUniversalForwarder\ etc\ apps\ Windows\ local
restart splunkforwarder:
splunk restart

it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.

[Ovi]

Well...I fixed it but I am no smarter than before.
The solution was to restart the Splunk server then all the new forwarders connected without issues

[sansay]

OK but what is your splunk server? Is it an indexer, a search head, or both?
I am encountering the same problem. But the difference might be that we added 4 new indexers. And then we deployed the config change for one of the forwarders to see the new indexers.
It sees them and forward some logs once in a while but I see a lot of those cooked connections errors.