Hi,
I upgraded a Search Head to 6.6.0, and am getting the following messages continuously...
5-10-2017 13:12:10.558 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:10.558 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:13.181 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:13.181 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:15.624 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:15.624 -0400 WARN HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
adding cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
parameter under [sslConfig]
in server.conf did the trick for us.
We had HTTP event collector servers stopped sending data once upgraded from v6.5 to v7.1.6.
put this in server.conf
[sslConfig]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
I've battled this issue so many times - nclancy, your comment was very helpful, however - I still had some issues.
At first, I opted to add the following to $SPLUNK_HOME/etc/system/local/inputs.conf:
[applicationsManagement]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
I believe there's a bug, because after a Splunk restart, the btool debug didn't report the change:
$ ./splunk btool inputs list --debug | grep cipher
/opt/splunkforwarder/etc/system/default/inputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
I ended up editing $SPLUNK_HOME/etc/system/default/inputs.conf and it did the trick. No more SSLv3 errors!
If you're at Splunk and can replicate this issue, I'm happy to provide a diag so we can address this bug.
Thanks!
I think your specific issue is actually that you should have edited the stanza [SSL]
, not [applicationsManagement]
.
Since your changes to default will be reverted upon upgrade, I highly recommend you try adding the stanza in local
again but as:
[SSL]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
Changes to the cipher suites between versions of splunk mean that OOTB the two versions of splunk will not have a common cipher to share the documentation advises providing a common cipher the two versions can agree on.
SSL/TLS are protocols - NOT ciphers. In particular, TLS is an evolution of SSL.
The relevant change is in $SPLUNK_HOM/etc/system/default/server.conf, and is the change to cipherSuite. In 6.4.1 this is set to
TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
and in 6.6.1 this is set to
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
TLSv1+HIGH (and corresponding for TLSv1.2) means all ciphers compatible with TLSv1 of HIGH strength. There is some overlap here with the ciphers compatible with SSL3.0. However, none of the SSL3.0 ciphers appear in the 6.6.1 list.
To see this more clearly, take a Linux system with openssl installed (almost any Linux system will do!).
Run:
openssl ciphers SSLv3+HIGH
openssl ciphers TLSv1+HIGH
Note that these give you the same results. However, they all end with SHA. In the explicit list you provide in 6.6.1 they all end with SHA, so it's easy to see that there's no overlap with SSLv3+HIGH and the new list in 6.6.1 - leading to the behaviour observed. Any system (such as Splunk 6.1) which only supports TLS1.0 and below (including SSL3) won't be able to communicate with a Splunk 6.6.1 server with default config only suitable for TLS1.2.
thank you nclancy, this was a fantastic help.
@a212830 - Would you accept this answer if it helped?
Looks like there's some known issues related to SSL and upgrades.
Do any of these items seem like the cause? http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues
Do you see any communication failures or just these warnings?
Include any SSL related conf from your server.conf or web.conf (probably good to use btool here). Since it was an upgrade, there's always a chance there was a leftover ssl config from a prior release that conflicts with modern security requirements for SSL.
I'm also experiencing this after upgrade from 6.5.1 to 6.6.1. Had to rollback to previous version and all worked.
Are you using older universal forwarders pre 6.2.x and sending traffic to a splunk tcp SSL port on the indexer?
In particular the older 6.0/6.1 series releases:
6.0.0 to 6.0.6 forwarders
6.1.0 to 6.1.4 forwarders
If so you can make the changes described in the known issues for 6.6.2 or upgrade your forwarders to a new version.
I suspect that your seeing older forwarders attempting to use an SSL/TLS cipher suite that is no longer supported by a modern version of the Splunk enterprise server.
any updates on this? Im experiencing the same issue now.
Its a bug ... Roll back to previous version and see
Everything works as normal