Splunk events are stopping. When we check the logs, we are seeing all of these errors. When we restart Splunk, it works for a few minutes and then goes away again. Here is the error message.
06-21-2016 22:20:11.986 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:20:19.396 +0100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1300 seconds.
06-21-2016 22:20:41.969 +0100 WARN TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:20:41.969 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:11.982 +0100 WARN TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:11.982 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:41.984 +0100 WARN TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:21:41.984 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
06-21-2016 22:21:59.507 +0100 WARN TcpOutputProc - Forwarding to indexer group default-autolb-group blocked for 1400 seconds.
06-21-2016 22:22:11.974 +0100 WARN TcpOutputFd - Connect to xx.xxx.x.xxx:9997 failed. No connection could be made because the target machine actively refused it.
06-21-2016 22:22:11.975 +0100 ERROR TcpOutputFd - Connection to host=xx.xxx.x.xxx:9997 failed
We have tried what previous users suggested by adding this to the inputs.conf to no avail.
[splunktcp://9997]
connection_host = none
Make sure you have this line in your iptable
cat /etc/sysconfig/iptables
vi /etc/sysconfig/iptables
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9997 -j ACCEPT
That is the default port for the forwarders to send (non-SSL) data to the Indexers:
https://answers.splunk.com/answers/118859/diagram-of-splunk-common-network-ports.html
Unless you changed something on purpose, it should "just work" right out of the box, if you have port 9997 opened for incoming connections from you forwarders to your indexers. This looks like a classic firewall deny scenario.