I reported this to Splunk Support at https://www.splunk.com/page/submit_issue as Case 107515. Seth Garvin's response was very helpful. For reference, I am working with Splunk forwarder version 5.0.1, build 143156, for Solaris 10, SPARC, from the file splunkforwarder-5.0.1-143156-SunOS-sparc.tar.Z.
There is a bug in Splunk 5.0.1, such that in the file /opt/splunkforwarder/etc/system/local/server.conf when the sslKeysfilePassword is set to the encrypted version of the special string "password", it is misread and incorrectly set. The successful work around was to edit the file /opt/splunkforwarder/etc/system/local/server.conf, and change the sslKeysfilePassword to the unencrypted string "password". Then do /opt/splunkforwarder/bin/splunk restart. After the restart, I notice that the order of the values in server.conf had changed, and the value for sslKeysfilePassword is again displayed encrypted.
An additional problem is that, because of another bug in Splunk 5.0.1, the results of the output of /opt/splunkforwarder/bin/splunk list forward-server can be wrong. As a consequence, the Splunk supplied gauge normally used to show the status of splunk forwarder systems displays erroneous info. Seth Garvin identifies this issue as known bug SPL-55793.
After the Splunk restart, /opt/splunkforwarder/bin/splunk list forward-server continues to report that the forwarder is Configured but inactive. This is false. Going to the Splunk indexer, and doing a search for the just restarted Splunk forwarder using host=forwarderhost yielded more than 1,000 events for the past 24 hours.
As a diagnostic of the failure to communicate via SSL from Splunk forwarder to Splunk indexer, for me, an indication that SSL is working on the Splunk indexer is the presence of this message in the Splunk log.
root@indexerhost# grep "port 9997 is reserved" /opt/splunk/var/log/splunk/splunkd.log | tail -1
01-07-2013 13:06:25.325 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
An indication that SSL is NOT working on the Splunk forwarder is the presence of these messages in the Splunk log.
root@forwarderhost# grep "SSL" /opt/splunkforwarder/var/log/splunk/splunkd.log
01-07-2013 13:10:32.279 -0500 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP DecryptFinal_ex:bad decrypt.
01-07-2013 13:10:32.279 -0500 ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server indexerhost.domain.com:9997
01-07-2013 13:10:32.282 -0500 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
... View more