Universal Fowarder does not send data to Splunk Forwarder, Indexer

[rb51]

Hi all,

Still new to Splunk management....

For some reason a Splunk Universal Forwarder (Windows) is not forwarding logs to my Splunk Forwarder and then the Splunk Indexer.

Universal forwarder (6.4.2) was installed successfully on a Windows 2008 R2 VM. Running netstat I can see that the connection between this server and the SPlunk Forwader is established on port 9997. Likewise on Splunk Forwarder server the netstat shows the connection esbalished. No Windows firewall on either server.

However on the splunkd.log file from the Windows Server (Universal Forwarder client) I can see the following message:

"Connection to host=SplunkForwarderIP:9997 failed. No connection could be made because the target machine actively refused it."
"Connect to SplunkForwarderIP:9997 failed"

Universal Forwarder and Splunk Forwarder are on different networks separated by a Layer3 switch. Traffic between these 2x networks have been completely open. Hence netstat shows connection established and telnet works fine.

The Splunk Forwarder then sends data to the Indexer on Local site and also to Indexer on DR site.

Splunk Forwarder server has been configured to receive data on 9997.

I am really struggling with this one, so would appreciate comments and suggestions.

Maybe next step is to install WireShark on SPlunkForwarder to capture the traffic and understand why it is refusing connection from UniversalForwarderClient VM.

UNIVERSAL FORWARDER conf files
inputs.conf (...etc\system\local)
[default]
host = testserver
[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0

outputs.conf (...etc\system\local)
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = SplunkIndexerIP:9997
[tcpout-server://SplunkIndexerIP:9997]

inputs.conf (...\etc\apps\Splunk_TA_windows\local)
[WinEventLog://Application]
[WinEventLog://Security]
disabled = 0
[WinEventLog://System]
disabled = 0

SPLUNK INDEXER conf files
outputs.conf
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 1

[tcpout-server://SplunkIndexerLocalSiteHostname:9997]

[tcpout:default-autolb-group]
disabled = false
server = SplunkIndexerLocalSiteHostname:9997,SplunkIndexerDRSiteHostname:9997
[tcpout-server://SplunkIndexerDRSiteHostname:9997]

[ddrillic]

A good place to start is at I can't find my data!