Getting Data In

ERROR TcpOutputFd - Read error. Connection reset by peer : splunkforwarder

sanaa
New Member

Hi ,

I am pretty much new to Splunk. I want to forward audit.log of one of my Linux servers to view in Splunk Web. For this, I did the following steps:

1) Upgraded version of splunkforwarder to 6.4.2
2) Modified inputs.conf and outputs.conf
3) Restarted Splunk

But i am getting below logs in splunkd.log. Please let me know how to see these audit.logs in Splunk Web. Am I missing any steps?

08-23-2016 10:37:56.325 +0000 INFO  WatchedFile - Will begin reading at offset=5111808 for file='/opt/zenoss/log/audit.log'.
08-23-2016 10:37:56.626 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:03.020 +0000 INFO  TailReader - Could not send data to output queue (parsingQueue), retrying...
08-23-2016 10:38:26.227 +0000 ERROR TcpOutputProc - Can't find or illegal IP address or Name: NONE
08-23-2016 10:38:26.228 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:38:56.231 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:26.235 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:39:38.909 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 100 seconds.
08-23-2016 10:39:56.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:26.227 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:40:56.216 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:18.525 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 200 seconds.
08-23-2016 10:41:26.211 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:41:56.198 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:26.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:56.200 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
08-23-2016 10:42:58.896 +0000 WARN  TcpOutputProc - Forwarding to indexer group splunkcloud blocked for 300 seconds.

Please help

0 Karma

woodcock
Esteemed Legend

Here are some things that hopefully you can change/disable that can get in the way:
FIPS
selinux
firewall (firewalld)
missing route
dns

0 Karma

sraymondg
New Member

I tried to disable FIPS on Splunk forwarder as it looks like FIPS is disabled on Splunk cloud or indexer also any forwarder with FIPS turned on will fail to be allowed to connect.

On the mis-configured forwarders disable FIPS and reboot.

Check FIPS is disabled with the next command:

cat /proc/sys/crypto/fips_enabled 0

https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/SecuringSplunkEnterprisewithFIPS

The Federal Information Processing Standard (FIPS) uses government-certified versions of some algorithms to meet regulatory guidelines.
It should not be considered a security enhancement by itself, and might potentially reduce performance on your system.
Enable FIPS if it is a regulatory requirement for your environment.
Splunk Enterprise and the Universal Forwarder use an embedded FIPS 140-2-validated cryptographic module.
Thus you need FIPS enabled and running on both the Forwarder side and the Indexer side

0 Karma

vasanthmss
Motivator

Check your indexer version .. indexer should be high or equal version.. if not here are the few steps to troubleshoot,

  1. check your outputs.conf -

indexer ip - wrong ips / firewall issue

  1. telnet the indexer ip from forwarder and check the connection is valid or not? use the below
    telnet

    eg:

    telnet 10.99.0.1 9997

hope this will helps you.

thanks,
V

V
0 Karma

justynap_ldz
Path Finder

@vasanthmss Do you have any other suggestions? 

We are working on Splunk 7.2.9.1. but encountered similar issue. 

ERROR TcpOutputFd - Read error. Connection reset by peer occured on one indexer. Splunkd stopped.

Then Splunk stopped on other 3 indexers that ended up with the following errors:
ERROR TcpOutputFd - Connection to host=xyzf failed and 
ERROR TcpOutputFd - Connect to host=xyzf refused. 

Also, in the same timeframe there was ClusterSlaveBucketHandler ERROR on one of the indexers.

Splunk version for all indexers is the same. I checked outputs.conf and run telnet between indexers. All fine.

Any hints will be much appreciated!

0 Karma

Sanjayr1081
Explorer

Hi,

I am having the same issue.

Logs are not going to index from forwarder and I am getting same error.

Did you got any solution for this? @justynap_ldz 

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...