Getting Data In

Universal Forwarder not sending data - timed out

john_byun
Path Finder

I've installed a universal forwarder on a linux box and configured it, but I'm getting the following errors. I'm running 5.0.1 and the indexer is currently listening on 9997:

From indexer:

11-05-2013 14:02:50.585 -0800 ERROR TcpInputProc - Error encountered for connection from src=xx.xx.xx.xx:60599. Timeout

From forwarder:
11-05-2013 20:23:49.189 -0500 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).

11-05-2013 20:23:49.475 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 20:24:06.849 -0500 ERROR AuthenticationManagerSplunk - Login failed. Incorrect login for user: admin

11-05-2013 20:37:09.152 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Detected connection to =xx.xx.xx.xx:9997 closed

11-05-2013 20:37:09.152 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 20:37:09.153 -0500 INFO TcpOutputProc - Closing stream for idx==xx.xx.xx.xx:9997

11-05-2013 20:37:29.621 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:37:49.155 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 20:42:09.152 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997

11-05-2013 20:50:39.168 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 20:50:39.168 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997

11-05-2013 20:51:00.107 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:55:39.110 -0500 WARN TcpOutputProc - Shutdown timed out for xx.xx.xx.xx:9997

11-05-2013 20:56:09.110 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 20:57:49.114 -0500 WARN TcpOutputProc - Connected to idx=xx.xx.xx.xx:9997. Not using ACK.

11-05-2013 21:03:09.116 -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Detected connection to xx.xx.xx.xx:9997 closed

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Will close stream to current indexer xx.xx.xx.xx:9997

11-05-2013 21:03:09.116 -0500 INFO TcpOutputProc - Closing stream for idx=xx.xx.xx.xx:9997

11-05-2013 21:03:29.601 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

11-05-2013 21:04:09.117 -0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out

Here is the configuration on the forwarder:

outputs.conf

[tcpout]
defaultGroup = default

[tcpout:default]
server = xx.xx.xx.xx:9997

[tcpout-server://xx.xx.xx.xx:9997]

hsesterhenn_spl
Splunk Employee
Splunk Employee

Well, old question but maybe worth to comment:

Remember to check you have a rule in inputs.conf somewhere.

Check this with

splunk btool inputs list --debug | less

and search for a stanza where there is NO "disable = 1" entry!

HTH,

Holger

0 Karma

kuido7Xdoc
Engager

add manually into file
opt/splunk/etc/system/local/inputs.conf

[splunktcp://9997]
disabled = 0

Akili
Path Finder

had the same problem, couldnt connect to indexer
in windows for universal forwarder installation ( 5.0.4) please check the files in:
path /SplunkUniversalForwarder/etc/system/local
replace the config files under with those from:
path /SplunkUniversalForwarder/etc/apps/Windows/local
restart splunkforwarder:
splunk restart

it should get connected
in splunk host i can see the forwarder has been connected and it has send logs. i had activated some advanced audit features.

lguinn2
Legend

What is the configuration on the indexer? Specifically, what is in the inputs.conf stanza that set up the tcpinput on 9997?

john_byun
Path Finder

Hmm...even though it was showing in the web gui, I couldn't find it in any of the inputs.conf files. I confirmed it was listening on 9997 using netstat.

In any case, I explicitly added it to my inputs.conf from the /splunk/etc/apps/search/local folder.

[splunktcp://9997]
connection_host = dns

I am still not seeing any data come in.

0 Karma

grijhwani
Motivator

Do you definitely have appropriate routing? Since you have redacted your source address it impossible to know if this is relevant. Are there firewalls intervening? Do they have rules to allow TCP on 9997 from source to indexer?

0 Karma

lguinn2
Legend

My first thought is that port 9997 is blocked. You should make sure that the port is open from the indexer to the forwarder.

0 Karma

john_byun
Path Finder

It's not being blocked. I can successfully telnet to port 9997 from the forwarder to the indexer.

Also, if it were blocked, I would not see the error message above from the indexer.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...