So I have been trying to get a solution where you do not need to have an SSL certificate on a universal forwarder for sending data base to Splunk on port 9997. However I can't seem to get it to work.
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf
and
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Outputsconf
This from the inputs.conf seems to indicate by default you do not need to have a SSL certificate on the forwarder at all
requireClientCert = <bool>
* Determines whether a client must present an SSL certificate to authenticate.
* Full path to the root CA (Certificate Authority) certificate store.
* The <path> must refer to a PEM format file containing one or more root CA
certificates concatenated together.
* Defaults to false.
My working conf with SSL certs on both ends.
inputs.conf
###############################
# Encrypted receiver
###############################
# Ref : http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousethedefault...
# Ref : http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcert...
[splunktcp-ssl:9997]
disabled = 0
# requireClientCert = false
#############
# IMPORTANT: If this configuration is placed within an local/inputs.conf in an app the password will not be encrypted.
# If you require encryption the inputs.conf definitions must be in $SPLUNK_HOME/system/local/inputs.conf
[SSL]
serverCert = $SPLUNK_HOME/etc/apps/cfgd_ssl_certs_servers/auth/myFullIndexerMediaCertificate.pem
sslPassword = *************
outputs.conf
# http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf
[tcpout]
defaultGroup = indexers
indexAndForward = false
[tcpout:indexers]
server = 192.168.55.55:9997
# IMPORTANT: Ensure indexer is also configured http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousethedefault... a configuration app is also available for that.
sslCertPath = $SPLUNK_HOME/etc/apps/cfgd_fwd_to_idx_ssl/auth/myFullForwarderCertificate.pem
sslPassword = *******
sslVerifyServerCert = false
# sslCommonNameToCheck = indexer.mydomain.com
# While compression should not add too much overhead if there are performance issues this could be disabled to see if this alieviates the problem.
useClientSSLCompression = true
Now trying this next configuration does not work. If I try to use the requireClientCert set to false (even though it defaults to false) I never see any data from the forwarder.
inputs.conf
###############################
# Encrypted receiver
###############################
# Ref : http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousethedefault...
# Ref : http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcert...
[splunktcp-ssl:9997]
disabled = 0
requireClientCert = false
#############
# IMPORTANT: If this configuration is placed within an local/inputs.conf in an app the password will not be encrypted.
# If you require encryption the inputs.conf definitions must be in $SPLUNK_HOME/system/local/inputs.conf
[SSL]
serverCert = $SPLUNK_HOME/etc/apps/cfgd_ssl_certs_servers/auth/myFullIndexerMediaCertificate.pem
sslPassword = ***********
outputs.conf
# http://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf
[tcpout]
defaultGroup = indexers
indexAndForward = false
[tcpout:indexers]
server = 192.168.55.55:9997
# IMPORTANT: Ensure indexer is also configured http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousethedefault... a configuration app is also available for that.
sslVerifyServerCert = false
# sslCommonNameToCheck = indexer.mydomain.com
# While compression should not add too much overhead if there are performance issues this could be disabled to see if this alieviates the problem.
useClientSSLCompression = true
So the takeaway here is what does the requireClientCert do if it doesn't work when a forwarder contacts the server without a certificate?
well, yes, on the inputs.conf examples also, they say "false" and specify both certificates.
the splunk SSL documentation very confusing one.
https://docs.splunk.com/Documentation/Splunk/6.6.1/Admin/Inputsconf
[SSL]
serverCert=$SPLUNK_HOME/etc/auth/server.pem
password=password
rootCA=$SPLUNK_HOME/etc/auth/cacert.pem
requireClientCert=false
Please check this, incase if you have read it already -
IMPORTANT NOTE ABOUT "requireClientCert" :
As of Splunk 4.2.4, setting "requireClientCert = true" in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false".
We have set "requireClientCert = true". This requires the following conditions to be met :
a) "rootCA" must point to a file containing the CA's public key. In our example, it's the myCACertificate.pem file we generated in step 1.
b) The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 4) is signed by that CA.
c) The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 4). This password is "server_privkey_password" in our example.
https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA
The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.
Granted I can see why it would be a safe guaranteed way to stop a imposter forwarder from sending in data.
That said though it appears Splunk is completely ignoring the requireClientCert because even when I set it to false it still doesn't work.