-0500 ERROR TcpOutputFd - Read error. Connection reset by peer. not not able to forward data into index

[rhirasin]

$ tail -f splunkd.log
06-19-2017 06:08:12.823 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:08:16.540 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:08:42.692 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:12.560 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:16.562 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:09:42.437 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:09:49.310 -0500 WARN TcpOutputProc - Forwarding to indexer group indexers blocked for 3500 seconds.
06-19-2017 06:10:12.308 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:10:16.583 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:10:42.177 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:11:12.050 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer
06-19-2017 06:11:16.606 -0500 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.207.140.100_8089_apsrs3355.uhc.com_apsrs3355_128161B5-DBCF-49EE-91B7-406544EB0CDA
06-19-2017 06:11:29.326 -0500 WARN TcpOutputProc - Forwarding to indexer group indexers blocked for 3600 seconds.
06-19-2017 06:11:41.924 -0500 ERROR TcpOutputFd - Read error. Connection reset by peer

[rhirasin]

my issue go resolved. I have to use certs file in my secure env.

thanks for your quick help

[arunkchow]

Our org has a small splunk setup. I am trying to secure the splunk with letsencrypt. I have the certs already and put them in /opt/splunk/etc/auth/certs path.
Lets encrypt issues the files as cert.pem, chain.pem, fullchain.pem and privkey.pem.
I pointed to the location of certs in both web.conf and server.conf under /opt/splunk/etc/system/local/ on indexer server and outputs.conf on forwarders.
But I am still getting the same error and forwarders don't forward any data.

[jkat54]

I converted your comment to answer. Please mark it as the answer.

[jkat54]

It looks like you're trying to send data to the indexer on port 8089. data input is usually on port 9997. Please make sure your specifying the correct forwarding port in your forwarder's outputs.conf.

[rhirasin]

here is my output file. its look good. but still its not working

cat outputs.conf

[tcpout]

whitelist all event forwarding

forwardedindex.0.whitelist = .*

null out the following splunk defaults

forwardedindex.1.blacklist =
forwardedindex.2.whitelist =

defaultGroup = indexers

[tcpout:indexers]

Add all indexers on the following line. Separate indexers with a comma, if more than one exists.

server = server1.uhc.com:9997,server2.uhc.com:9997

Test Test

server = apsrd7043:9997

[jkat54]

run this command on the forwarder and post the details please.

./splunk btool outputs list --debug

[jkat54]

This is the indexer giving the error in the log:

apsrs3355

Yet you have apsrd7043 in the outputs you posted. So the btool command will show us which outputs.conf is pointing to apsrs3355, and then we can assist you on how to solve it... might be as simple as removing the other outputs.conf that points to apsrs3355.