Why am I receiving an error when deploying a new Splunk forwarder?

[gchotlineinfo]

Hi,

I try to deploy a new forwarder since i've updated my indexer to 7.0.3. I got some problems and i found my answers on this forum.
But I haven't been able to solve, below the error message in the splunkd.log

04-13-2018 13:22:44.069 +0000 INFO  TcpOutputProc - Removing quarantine from idx=IPAddress:9997
04-13-2018 13:22:44.072 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
04-13-2018 13:22:44.074 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
04-13-2018 13:22:44.074 +0000 WARN  TcpOutputProc - Applying quarantine to ip=IPAddress port=9997 _numberOfFailures=2
04-13-2018 13:22:51.491 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:22:51.503 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:23:51.505 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:23:51.517 +0000 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_IPAddress_8089_Hostname_ShortName_E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 13:24:17.921 +0000 WARN  TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group splunkssl has been blocked for 600 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

And on my indexer :

04-13-2018 15:24:50.665 +0200 INFO  ClientSessionsManager:Listener_AppEvents - Received count=1 AppEvent from DC ip=172.25.225.49 name=E4BC416F-983F-4CEF-AA47-45BA28ED0FF3
04-13-2018 15:26:42.372 +0200 ERROR TcpInputProc - Error encountered for connection from src=IPAddress:47781. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Port 8089, 9997 listen and telnet in between works.
Forwarder outputs.conf

[tcpout]

[tcpout:splunkssl]
server = indexer:9997

[tcpout-server://indexer:9997]
sslCertPath = /opt/splunkforwarder/etc/certs/splunk-sys-forwarder.pem
sslCommonNameToCheck = indexer
sslPassword = CaCertPassword
sslRootCAPath = /opt/splunkforwarder/etc/certs/cacert.pem
sslVerifyServerCert = false

Indexer inputs.conf

[splunktcp-ssl:9997]
disabled = 0
connection_host = ip

[SSL]

serverCert = /opt/splunk/etc/certs/splunk-sys-indexer.pem
sslPassword = CaCertPassword
requireClientCert = false

[mkolkebeck]

I'd recommend putting your ssl settings in outputs.conf under your [tcpout:splunkssl]. Per the spec, the [tcpout-server://indexer:9997] stanza is optional, unless you need common name checking of a single instance across a distributed indexer deployment.

It's also possible that you may have an invalid sslPassword or bad certificate.

You should also verify that you can connect via s_client:

./splunk cmd openssl s_client -connect indexer:9997

[afroz]

splunk forwarders version must be equal or lower than indexers. Fix that problem, this error won't come.

[bcyates]

I downvoted this post because it is not true, per splunk docs

[mkolkebeck]

Per the link below, it's a best practice to have a higher indexer version, but not required.
http://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Compatibilitybetweenforwardersandind...

[s2_splunk]

Error encountered for connection from src=IPAddress:47781. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

Indicates that your forwarder is trying to use an SSL version not supported by your indexer. What version did you upgrade from on your indexer and what version is your forwarder?
As of 6.6 we will default to TLS1.2 and if your forwarder requests a lower SSL version you will see this message. Review the docs to see if the workaround works for you; or upgrade your UF to a version post 6.6.

[gchotlineinfo]

I upgraded from 6.2.2 to 7.0.3 for indexer and forwarders. I checked with the command : /opt/splunk/bin/splunk cmd btool inputs list --debug

/opt/splunk/etc/system/default/inputs.conf sslVersions = tls1.2

And on forwarder :

/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2

[tlam_splunk]

Please check the cipherSuite parameter and see they are matched in Indexer and forwarder

[gchotlineinfo]

Indeed, there is a difference :

Indexer

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

Forwarder

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256

I added these parameters on both sides, I have the same result.

sslVersions = tls1.2
cipherSuite = AES256-SHA256:DHE-RSA-AES256-SHA256