Getting Data In

How to troubleshoot error "Connection to host=xx.xxx.x.xxx:9997 failed"?

johnbuhlhiscox
New Member

Splunk events are stopping. When we check the logs, we are seeing all of these errors. When we restart Splunk, it works for a few minutes and then goes away again. Here is the error message.

06-21-2016 22:20:11.986 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:20:19.396 +0100 WARN  TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1300 seconds.
06-21-2016 22:20:41.969 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:20:41.969 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:11.982 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:11.982 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:41.984 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:41.984 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:59.507 +0100 WARN  TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1400 seconds.
06-21-2016 22:22:11.974 +0100 WARN  TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:22:11.975 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed

We have tried what previous users suggested by adding this to the inputs.conf to no avail.

[splunktcp://9997]
connection_host = none
0 Karma

dkingfx
New Member

Make sure you have this line in your iptable
cat /etc/sysconfig/iptables
vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9997 -j ACCEPT

0 Karma

woodcock
Esteemed Legend

That is the default port for the forwarders to send (non-SSL) data to the Indexers:
https://answers.splunk.com/answers/118859/diagram-of-splunk-common-network-ports.html

Unless you changed something on purpose, it should "just work" right out of the box, if you have port 9997 opened for incoming connections from you forwarders to your indexers. This looks like a classic firewall deny scenario.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...