Why am I getting an SSL Unknown Protocol Error trying to receive data from a universal forwarder on my indexer?

[devinmclean]

I'm getting the following error when trying to receive data from a universal forwarder on my indexer:

(log from indexer's splunkd.log, replaced IP with 192.168.1.10's):

08-04-2015 17:20:13.046 -0400 ERROR TcpInputProc - Error encountered for connection from src=192.168.1.10:37735. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

(log from UF's splunkd.log):

08-04-2015 17:18:09.453 -0400 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eInit for 192.168.1.10:9997
08-04-2015 17:18:09.453 -0400 DEBUG TcpOutputProc - tcpConnect to 192.168.1.10:9997
08-04-2015 17:18:09.453 -0400 DEBUG TcpOutputProc - ConnectionSuccessful. _rawConnectionState=eRawTcpConnectInProgress
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - ConnectionSuccessful. _cookedVersionNegState=eV3TcpConnectInProgress
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eV3ConnectInit for 192.168.1.10:9997
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - v3Connect to 192.168.1.10:9997
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - AsyncV3Connector::getData for eSendSignature
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - write Successful for eSendSignatureInProgress
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - AsyncV3Connector::getData for eSendCapability
08-04-2015 17:18:09.454 -0400 DEBUG TcpOutputProc - Connector::connectionFailed
08-04-2015 17:18:09.455 -0400 DEBUG TcpOutputProc - Connector::cookedConnectionFailed
08-04-2015 17:18:09.455 -0400 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eFailed for 192.168.1.10:9997

[dwaddle]

This sounds like a configuration error of some sort. Like one end is expecting TLS and the other is sending in cleartext, or there is a disagreement on configuration of things like the sslVersions and supportSSLV3Only. I would double check configurations on both ends and make sure they agree.

[devinmclean]

Turned on debug logging for TcpOutputFd and am now see these errors:

08-10-2015 13:12:55.904 -0400 ERROR TcpOutputFd - Read error. Connection reset by peer
08-10-2015 13:12:55.904 -0400 DEBUG TcpOutputFd - after_eof detected

[devinmclean]

This is the most confusing part. I am having no problems with my original Indexers receiving data from forwarders. The only difference between the indexers are that the original ones were installed last year on version 6.1, then upgraded to 6.2.1, and then finally to 6.2.3 recently. The new ones were installed initially with 6.2.3. Maybe there's a key difference since one set had an upgrade path to 6.2 an the others were installed with 6.2.

[MuS]

Hi devinmclean,

according to the docs http://docs.splunk.com/Documentation/Splunk/6.2.4/Admin/Inputsconf compression is turned on by default on the indexer inputs.conf :

[splunktcp-ssl:<port>]
* Use this stanza type if you are receiving encrypted, parsed data from a forwarder.
* Compression for SSL is enabled by default.

On the forwarder http://docs.splunk.com/Documentation/Splunk/6.2.4/Admin/Outputsconf it defaults to a server.conf setting:

useClientSSLCompression = [true|false]
* Enables compression on SSL.
* Defaults to value of useClientSSLCompression from [sslConfig] stanza in server.conf.

and on the forwarder http://docs.splunk.com/Documentation/Splunk/6.2.4/Admin/Serverconf in the server.conf it defaults to on:

[sslConfig]
useClientSSLCompression = true|false
    * Turns on HTTP client compression. 
    * Server-side compression is turned on by default; setting this on the client side enables 
      compression between server and client.  
    * Enabling this potentially gives you much faster distributed searches across multiple Splunk instances.
    * Defaults to true.

Verify the settings and beside of this I don't know any other cause for this.

cheers, MuS

[devinmclean]

thanks for the research. I don't think this is the reason, because this forwarder is sending successfully to three other indexers, which would indicate that the default values are set and are working as intended. i've recently purchased four new indexers and am trying to configure them for receiving data, which is when i've run into this issue.

[MuS]

Another idea, check if you're using [tcp-ssl:] instead of [splunktcp-ssl:] in the indexers inputs.conf

[devinmclean]

I am using [splunktcp-ssl://9997]

[MuS]

out of ideas in this case, sorry

[devinmclean]

I read that the compressed flag has no effect on SSL-enabled data transfer between forwarders and indexers. Either way: yes, I have checked for enabling compressed = true on indexer and forwarder.

[MuS]

HeHe, true copy/past error - it should be useClientSSLCompression and sorry, I wasn't clear enough. What I meant was, has one the option enabled and the other disabled? This could lead to such errors.

[devinmclean]

I didn't supply a useClientSSLCompression option in either configuration file, so I'm hoping that they're using the default options. Should I try and set this field? Do you know of any other reason why this error could be happening?

[MuS]

Did you check if either side has compressed = true set in outputs.conf on the forwarder or inputs.conf on the indexer?