Hey all,
I'm having a really tough time getting my forwarders and indexer to talk over SSL using a non-default CA. I've searched through other answers, but haven't found any resolution. I've been following this guide:
https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA
I've also read through http://docs.splunk.com/Documentation/Splunk/6.2.3/Security/ConfigureSplunkforwardingtousesignedcerti....
I generated the CA root cert, generated & signed my CA public cert (myCACertificate.pem), then generated, signed, and my server cert with the public, private, and CA certs (myServerCertificate.pem) with "splunkserver.internal.domain"
I modified my inputs.conf:
# cat /opt/splunk/etc/system/local/inputs.conf
[default]
host = splunkserver
[splunktcp-ssl:9997]
compressed = true
disabled = 0
[SSL]
password = {myServerPrivateKey.key password}
rootCA = $SPLUNK_HOME/etc/certs/myCACertificate.pem
serverCert = $SPLUNK_HOME/etc/certs/myServerCertificate.pem
I restarted Splunk and see this in my splunkd.log, which supposedly means the server has accepted the certificate:
09-10-2015 17:12:30.560 +0000 INFO loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
09-10-2015 17:12:30.879 +0000 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
I copied myCACertificate.pem and myServerCertificate.pem using SCP to the forwarder & modified its outputs.conf:
# cat /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
compressed = true
server = {IP_of_server}:9997
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = {myServerPrivateKey.key password}
sslRootCAPath = /opt/splunkforwarder/etc/certs/myCACertificate.pem
sslVerifyServerCert = true
I restarted the Splunk forwarder and get this output:
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
Stopping splunk helpers...
Done.
Splunk> Winning the War on Error
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=587690100 error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error.
Couldn't initialize SSL Context for HTTPClient in ServerConfig
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
I see this in splunkd.log:
09-10-2015 17:32:10.531 +0000 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=587690100 error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error.
09-10-2015 17:32:10.532 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
09-10-2015 17:32:10.532 +0000 ERROR HTTPServer - SSL will not be enabled
I have a few questions:
Why is the Splunk forwarder trying to use /opt/splunkforwarder/etc/auth/server.pem
when I specified /opt/splunkforwarder/etc/certs/myServerCertificate.pem
? I see that according to the guide, the expected logs from the forwarder use server.pem as well.
Should I be able to run openssl x509 -in /opt/splunkforwarder/etc/certs/myServerCertificate.pem -text -noout
and get output without entering a password?
and finally... what the heck am I doing wrong?!
Any help is appreciated! Thanks!
I figured this out. When I created my certificates, I used the same CN for the CA and server certs. I used a different name for my CA and that worked.
I figured this out. When I created my certificates, I used the same CN for the CA and server certs. I used a different name for my CA and that worked.
I answered my second question - the correct command is:
openssl rsa -in /opt/splunkforwarder/etc/certs/myServerCertificate.pem -text
I am able to decrypt that key using the same password I entered in the outputs.conf on the forwarder.