Security

OpenSSL sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)

iiierdna
Explorer

I am setting up SSL between a Heavy Forwarder to Indexer in AWS but I am encountering an error. In this SSL setup, I am using self-signed root CA. And I am using Splunk v6.5.2 on both Heavy Forwarder and Indexer.
Tested a non-SSL connection between the Heavy Forwarder to Indexer works fine. Performing telnet to server and port 9997 works fine as well.

The setup on the Indexer in AWS to listen SSL port 9997 are showing that the setup has been successful,

03-01-2017 13:36:42.317 +0900 INFO  TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
03-01-2017 13:36:42.317 +0900 INFO  TcpInputConfig - IPv4 port 9997 will negotiate new-s2s protocol
03-01-2017 13:36:42.319 +0900 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL

Then I proceeded to setup the Heavy Forwarder but showed the following error,

03-01-2017 11:12:14.228 +0800 ERROR TcpOutputFd - Connection to host=1.1.1.2:9997 failed. sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)

And so I switched on debug mode to see the error on the Heavy Forwarder,

03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - ---- existing clients - start ----
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - ---- existing clients - end ----
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - AutoLB timer started to select new connection
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - BEGIN - randomizeConnectionsList
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Indexer uri 1.1.1.2:9997, client refCount=0, client=NULL
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - BEGIN - After sorting
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Indexer uri 1.1.1.2:9997, client refCount=0, client=NULL
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Found a candidate indexer which is currently not connected. 1.1.1.2:9997, client refCount=0, client=NULL
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - getting connected clients
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eInit for 1.1.1.2:9997
03-01-2017 12:37:20.147 +0800 DEBUG TcpOutputProc - tcpConnect to 1.1.1.2:9997
03-01-2017 12:37:20.148 +0800 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
03-01-2017 12:37:20.236 +0800 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...
03-01-2017 12:37:20.237 +0800 ERROR TcpOutputFd - Connection to host=1.1.1.2:9997 failed. sock_error = 10054. SSL Error = error:00000000:lib(0):func(0):reason(0)
03-01-2017 12:37:20.237 +0800 DEBUG TcpOutputProc - Connector::connectionFailed
03-01-2017 12:37:20.237 +0800 DEBUG TcpOutputProc - Connector::cookedConnectionFailed
03-01-2017 12:37:20.237 +0800 DEBUG TcpOutputProc - Connector::runCookedStateMachine in state=eFailed for 1.1.1.2:9997
03-01-2017 12:37:20.237 +0800 DEBUG TcpOutputProc - Cannot find any valid descriptors when looking for new indexer.
03-01-2017 12:37:20.237 +0800 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...

At this point I am not sure what exactly is the problem. I have attempted various methods from other posts without any success.
Reference sites:

Subsequently,
I have tried to go into openssl to attempt a connection,

OpenSSL > s_client -connect 1.1.1.2:9997
CONNECTED<000000C4>
write:errno=10054
 ---
no peer certificate available
 ---
No client certificate CA names sent
 ---
SSL handshake has read 0 bytes and written 306 bytes
 ---
New, <NONE>, Cipher is <NONE>
Secure Renegotiation is NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
         Protocol    : TLSv1.2
         Cipher       : 0000
         Session-ID:
         Session-ID-ctx:
         Master-Key:
         Key-Arg      : None
         PSK identity: None
         PSK identity hint: None
         SRP username: None
         Start Time: 1488532303
         Timeout       : 300 <sec>
         verify return code: 0 <ok>
 ---
error in s_client
OpenSSL>

Anyone know what could be the issue here?

Tags (1)

iiierdna
Explorer

UPDATE: This issue has been resolved.

TL;DR A NG firewall was performing deep packet inspection on port 9997 to filter out SSL connection. Created a separated rule to allow SSL for port 9997

We did various testing like setting up a separate instance and install Heavy Forwarder alone within the same network as the non-working Heavy Forwarder to isolate whether was it the instance problem or the network problem. The test results concludes that there was something blocking SSL connection going out to the indexer on AWS.

lim2
Communicator

Thanks for the information innerdna, that it was the fix for my issue too.

0 Karma

jtacy
Builder

Your tests suggest that your indexer is not configured for SSL receiving. You should have an input stanza that looks like this in the inputs.conf of the indexer:

[splunktcp-ssl:9997]

Are you sure that you don't have an ordinary splunktcp stanza somewhere? When you restart the indexer, you should get output like this in splunkd.log (note the two references to SSL):

03-05-2017 17:12:10.455 -0600 INFO  TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
03-05-2017 17:12:10.455 -0600 INFO  TcpInputConfig - IPv4 port 9997 will negotiate new-s2s protocol
03-05-2017 17:12:10.456 -0600 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL
0 Karma

iiierdna
Explorer

My apologies for the confusion, I have updated my post to reflect the correct log entries.
Re-tested the SSL again, And it does show that it is listening at 9997 (SSL), but I am getting the same error.

0 Karma

jtacy
Builder

OK, super! Even if you have client authentication enabled on the indexer, openssl s_client should at least return the server certificate. Do you get a different result if you try the s_client test on the AWS host itself? If you get the same result, can you post your inputs.conf and any relevant log entries from splunkd.log, both from the indexer? In my testing, it either works or I get a connection refused; your situation seems to be in the middle so it's interesting.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...