Hi. I have a Deployment Server, and I am using it to configure a Deployment Client, which is a Windows Universal Forwarder. I want to use SSL for both Forwarder-Indexer communications, as well as Deployment Client - Deployment Server communications, using "our" Certs (not Splunk's).
The initial state is that Forwarder-Indexer communication, as well as Deployment Client - Deployment Server communication, is working fine, but not using our Certs.
My first step is to configure the Forwarder-Indexer communication to use our Certs. So, I modify outputs.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. It works like a charm - the UF is now sending data to the Indexers, using SSL, with our Certs.
So now I want to make the equivalent change for the Deployment Client - Deployment Server communication. This time, I modify server.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. This does NOT work. splunkd.log, on both Deployment Client and Deployment Server, report errors (shown below), and the two endpoints no longer communicate, and a "./splunk list deploy-clients" command on the Deployment Server no longer displays the target machine as one of its Deployment Clients.
As best as I can tell, I am using the same "forwarder.pem", "cacert.pem", and cert passwords in the two .conf files (outputs and server), but things just aren't working for the Deployment Client - Deployment Server communications.
Any enlightenment, or at least tips on troubleshooting this, would be GREATLY appreciated!
Here are the splunkd.log errors...
09-21-2011 20:38:02.971 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
09-21-2011 20:38:02.971 -0700 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
09-21-2011 20:38:03.994 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL will not be enabled
09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error for fd from HOST:*hostname*, IP:*ip-addr*, PORT:50252
And, here are the .conf files...
[tcpout]
defaultGroup = splunkssl-autolb-group
disabled = false
[tcpout-server://*hostname*:9998]
sslCertPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem
sslPassword=*password*
sslRootCAPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\cacert.pem
sslVerifyServerCert=true
sslCommonNameToCheck=*full_hostname*
altCommonNameToCheck=*short_hostname*
there is more, but not SSL-related
[sslConfig]
enableSplunkdSSL = true
sslKeysfile = forwarder.pem
sslKeysfilePassword = *password*
caCertFile = cacert.pem
caPath = C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local
I fear that the problem here is that on first time run, splunkd encodes the default password for the default server certificate used for the splunkd management port, and outputs that to $SPLUNK_HOME/etc/system/local/server.conf :
# cat etc/system/local/server.conf
[sslConfig]
sslKeysfilePassword = $1$S6+KkTowacjF
As you know, settings in etc/system/local will take precedence over any other instance of the same configuration key anywhere else. Which means that the sslKeysfilePassword that you set with deployment server will not be taken into account.
You can correct this by deleting the sslKeysfilePassword entry in etc/system/local/server.conf and restarting splunkd, but this is not something that deployment client can do for you.
hexx, others,
This is EXACTLY what I concluded as well, after careful troubleshooting: My password value, in my app version of server.conf, was being trumped by the etc/system/local version.
So, I first tried fixing this with the following Install sequence...
1) Install via msiexec command, with the LAUNCHSPLUNK=0 flag, to prevent Splunk from starting and creating server.conf.
2) Create, and copy files to, my app directory, including a version of server.conf with MY certs and password.
3) Start Splunk.
Unfortunately, this did NOT work. I found that, even with the LAUNCHSPLUNK=0 flag, splunk creates etc/system/local/server.conf at Install time, with ITS password, which, as you note, will override anything I put into my app directory.
I was going to try one more sequence...
1) Install via msiexec command, with the LAUNCHSPLUNK=0 flag, to prevent Splunk from starting and creating server.conf.
2) DELETE etc/system/local/server.conf.
3) Create, and copy files to, my app directory, including a version of server.conf with MY certs and password.
4) Start Splunk.
But I decided to "Ask Splunk" first, since it sure didn't feel like a "Best Practice".
If the above Install sequence works, is that what is recommended? Is there a cleaner, more elegant approach? This one feels a bit "klooge"y. We are developing an automated process to deploy all our Windows forwarders, and we want it to be as solid as reasonably possible.
Thx again.
mfeeny1
That's what I decided to do as well. At least on linux, I can script this.
Here are the important bits for anyone who stumbles onto this post in future (work in progress):
# Repack Splunk UF installer to include needed files
# (passwd, splunk.secret, splunk-launch.conf...)
cd /tmp
FINALFOLDER="splunk_UF_install"
SOURCEFILES="splunk_UF_source_files"
SPLUNKUF=$(ls splunkforwarder-*.tgz)
# unpack Splunk UF
if [ -f $SPLUNKUF ]; then
# Prep staging folder
if [ -d $FINALFOLDER ]; then
rm -rf $FINALFOLDER
fi
mkdir $FINALFOLDER
cp $SPLUNKUF $FINALFOLDER/
tar -xf /tmp/$FINALFOLDER/$SPLUNKUF -C /tmp/$FINALFOLDER/
rm /tmp/$FINALFOLDER/$SPLUNKUF
echo "[OK] extraction of Splunk UF from $SPLUNKUF successful"
else
echo "[ERROR] extraction of Splunk UF failed ($SPLUNKUF does not exist)"
exit 1
fi
# copy over needed files
cp /tmp/$SOURCEFILES/passwd /tmp/$FINALFOLDER/splunkforwarder/etc/
cp /tmp/$SOURCEFILES/splunk.secret /tmp/$FINALFOLDER/splunkforwarder/etc/auth/
cp /tmp/$SOURCEFILES/splunk-launch.conf /tmp/$FINALFOLDER/splunkforwarder/etc/
cp -r /tmp/$SOURCEFILES/my_targetDeploymentServer_app /tmp/$FINALFOLDER/splunkforwarder/etc/apps/
cp -r /tmp/$SOURCEFILES/my_ssl_config_app /tmp/$FINALFOLDER/splunkforwarder/etc/apps/
touch /tmp/$FINALFOLDER/splunkforwarder/etc/splunk.version
echo "[OK] modifications DONE"
# tar the splunkforwarder dir
cd /tmp/$FINALFOLDER && tar -czf splunkforwarder.tgz splunkforwarder && cd /tmp
rm -rf /tmp/$FINALFOLDER/splunkforwarder
echo "[OK] splunkforwarder compressed successfully"
# Add instructions and final tar
cp /tmp/$SOURCEFILES/install_instructions.txt /tmp/$FINALFOLDER/
tar -czf $FINALFOLDER.tgz $FINALFOLDER/splunk_UF_install_instructions.txt $FINALFOLDER/splunkforwarder.tgz
echo "[OK] overall package created successfully"
# Cleaning up
rm -rf /tmp/$FINALFOLDER
rm -rf /tmp/$SOURCEFILES
echo "[OK] Cleanup done. Exiting..."
exit 0
Do the encoded representations for server.conf/sslKeysfilePassword and outputs.conf/sslPassword match? To verify this, compare the output of "%SPLUNK_HOME%\bin\splunk cmd btool server list sslConfig --debug" and "%SPLUNK_HOME%\bin\splunk cmd btool outputs list tcpout-server --debug"