Along the lines of this question - http://splunk-base.splunk.com/answers/11577/splunk-overwrites-outputsconf-and-inputsconf-on-reboot, but I've tried a few additional things I'd like to note.
To reiterate the issue - I am trying to enable SSL in my outputs.conf for one of my forwarders. I am using the default certs and the default password "password".
I've changed the location of my outputs.conf quite a few times, trying these paths:
1) ./etc/system/local/outputs.conf
2) ./etc/apps/forwarder/local/outputs.conf
3) ./etc/apps/forwarder/default/outputs.conf
No matter which of these paths I choose, I continue to run into this pattern:
1) Update outputs.conf to have a sslPassword of "password"
2) Use btool to check outputs and note the sslPassword:
./bin/splunk cmd btool outputs list --debug
forwarder [tcpout]
system autoLB = true
forwarder defaultGroup = splunk
system forwardedindex.0.whitelist = .*
system forwardedindex.1.blacklist = _.*
system forwardedindex.2.whitelist = _audit
system forwardedindex.filter.disable = false
system maxQueueSize = 500KB
forwarder [tcpout-server://server.domain:8002]
forwarder sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
forwarder sslPassword = password
forwarder sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
forwarder [tcpout:splunk]
forwarder compressed = true
forwarder disabled = false
forwarder server = server.domain:8002
3) Restart splunkd - ./bin/splunk restart splunkd
4) Use btool to check outputs again and note the sslPassword again:
./bin/splunk cmd btool outputs list --debug
forwarder [tcpout]
system autoLB = true
forwarder defaultGroup = splunk
system forwardedindex.0.whitelist = .*
system forwardedindex.1.blacklist = _.*
system forwardedindex.2.whitelist = _audit
system forwardedindex.filter.disable = false
system maxQueueSize = 500KB
forwarder [tcpout-server://server.domain:8002]
forwarder sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
forwarder sslPassword = $1$ZMTWcdnuueG6
forwarder sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
forwarder [tcpout:splunk]
forwarder compressed = true
forwarder disabled = false
forwarder server = server.domain:8002
The sslPassword has been hashed, just as it mentions here - http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_DefaultCerts in the note for #2.
Note that the server certificate pass
phrase will be hashed and stored in
$SPLUNK_HOME/etc/system/local/outputs.conf,
overwriting the clear-text value of
"sslPassword" if it was defined there.
If "sslPassword" was defined in
clear-text in an outputs.conf located
in an app, it will not be hashed
there and will still be present in
clear text in that location. This
doesn't matter too much in this case
since the pass phrase for the default
server certificate is well known.
The note claims that the password will not be hashed if located in an app. But it is hashed when I used any of the locations above, where I consider paths 2 and 3 to be an app path.
I don't have an issue with the hashing, but I feel that it has to do with the SSL error I am getting:
ERROR SSLCommon - Can't read key file
$SPLUNK_HOME/etc/auth/server.pem
errno=101077092 error:06065064:digital
envelope
routines:EVP_DecryptFinal_ex:bad
decrypt.
When I check the password of the server.pem file using openssl:
openssl rsa -in /logs/splunk_forwarder/etc/auth/server.pem -text
Enter pass phrase for /logs/splunk_forwarder/etc/auth/server.pem:
I enter "password"
It works.
So either, the hashing needs to stop or needs to work.
Doing splunk add forward-server <host:port> -ssl-cert-path /path/ssl.crt -ssl-root-ca-path /path/ca.crt -ssl-password <password>
as described at http://docs.splunk.com/Documentation/Splunk/4.2.4/Deploy/Deployanixdfmanually works. Then you can move etc/local/{outputs,server}.conf (which contain the hashed password) to app dirs if desired, and restart.
Doing splunk add forward-server <host:port> -ssl-cert-path /path/ssl.crt -ssl-root-ca-path /path/ca.crt -ssl-password <password>
as described at http://docs.splunk.com/Documentation/Splunk/4.2.4/Deploy/Deployanixdfmanually works. Then you can move etc/local/{outputs,server}.conf (which contain the hashed password) to app dirs if desired, and restart.
This is because the default password is in the "system/default/"
And at start splunk encrypts it and save to "system/local/"
Now the seed used for the encryption can be different on every instance, therefore the encrypted password is different. A solution to avoid it is to uniformize the seed when installing ($SPLUNK_HOME\etc\auth\splunk.secret), or use the previous method.
Surprisingly - using the CLI vs modifying the files directly does work. Thank you.