Before restart:
on the forwarder:
$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = ssl_group.domain.com_9997
[tcpout:ssl_group.domain.com_9997]
server = tico:9997
[tcpout-server://tico:9997]
sslCertPath=$SPLUNK_HOME/etc/auth/server.pem
sslPassword=password
sslRootCAPath=$SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert=false
and the indexer:
$ sudo cat /opt/splunk/etc/system/local/inputs.conf
[SSL]
serverCert=$SPLUNK_HOME/etc/auth/server.pem
password=password
rootCA=$SPLUNK_HOME/etc/auth/cacert.pem
requireClientCert=false
[splunktcp-ssl:9997]
and after the restart of both forwarder and indexer
on the forwarder:
$ sudo cat /opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = ssl_group.domain.com_9997
[tcpout:ssl_group.domain.com_9997]
server = tico:9997
[tcpout-server://tico:9996]
sslPassword = $1$gfv0UiKu6wR2
[tcpout-server://tico:9997]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = $1$gfv0UiKu6wR2
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem
sslVerifyServerCert = false
and on the forwarder:
$ sudo cat /opt/splunk/etc/system/local/inputs.conf
[splunktcp-ssl:9997]
[SSL]
password = $1$nZo1AKEa4vCH
requireClientCert = false
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
And when testing a quick "logger" command, I see nothing from the forwarder showing up on the indexer's search.
If I get nasty and do:
$ sudo chattr +i /opt/splunk/etc/system/local/outputs.conf
and
$ sudo chattr +i /opt/splunk/etc/system/local/inputs.conf
respectively on the forwarder and indexer, I can see splunk trying to overwrite my configs in the log:
02-11-2011 09:47:44.357 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1
attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1]
02-11-2011 09:47:44.357 ERROR PropertiesMapConfig - Failed to save settings: /nobody/system/outputs/tcpout-server://tico:9996 (user: , app: , root: /opt/splunk/etc): Could
not flush changes to disk: /nobody/system/outputs/tcpout-server://tico:9996/sslPassword: ConfPathMapper: /opt/splunk/etc/system/local
02-11-2011 09:47:44.357 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-11-2011 09:47:44.357 INFO TcpOutputProc - initializing single connection with retry strategy for tico:9996
02-11-2011 09:47:44.381 INFO TcpOutputProc - attempting to connect to tico:9996...
02-11-2011 09:47:44.381 INFO TcpOutputProc - Will retry at max backoff sleep forever
02-11-2011 09:47:44.381 INFO TcpOutputProc - Using SSL for server tico:9997, sslCertPath=/opt/splunk/etc/auth/server.pem
02-11-2011 09:47:44.382 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1
attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1]
02-11-2011 09:47:44.382 WARN PropertiesMapConfig - Error removing preexisting values: /nobody/system/outputs/tcpout-server://tico:9997 [ConfPathMapper: /opt/splunk/etc/sy
stem/local]
02-11-2011 09:47:44.382 WARN TcpOutputProc - Failed to make a connection, will retry.
02-11-2011 09:47:44.384 ERROR IniFile - Cannot rename /opt/splunk/etc/system/local/outputs.conf.tmp.30860 to /opt/splunk/etc/system/local/outputs.conf: Rename failed in 1
attempt(s) made between Fri Feb 11 09:47:44 2011 and Fri Feb 11 09:47:44 2011 [status code: 1]
02-11-2011 09:47:44.384 ERROR PropertiesMapConfig - Failed to save settings: /nobody/system/outputs/tcpout-server://tico:9997 (user: , app: , root: /opt/splunk/etc): Could
not flush changes to disk: /nobody/system/outputs/tcpout-server://tico:9997/sslVerifyServerCert: ConfPathMapper: /opt/splunk/etc/system/local
02-11-2011 09:47:44.384 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher=
02-11-2011 09:47:44.384 INFO TcpOutputProc - initializing single connection with retry strategy for tico:9997
02-11-2011 09:47:44.384 INFO TcpOutputProc - attempting to connect to tico:9997...
02-11-2011 09:47:44.384 INFO loader - Instantiated plugin: controlqueueoutputprocessor
02-11-2011 09:47:44.384 INFO PipelineComponent - Pipeline merging disabled in *mode.conf file
02-11-2011 09:47:44.384 INFO PipelineComponent - Pipeline typing disabled in *mode.conf file
02-11-2011 09:47:44.384 INFO loader - Running....
02-11-2011 09:47:44.384 INFO PipelineComponent - Launching the pipelines.
02-11-2011 09:47:44.384 ERROR IndexProcessor - received event for unconfigured/disabled index='_audit' with source='source::audittrail' host='host::localhost' sourcetype='
sourcetype::audittrail'
02-11-2011 09:47:44.385 WARN pipeline - Exiting pipeline tail gracefully: got eExit from processor tail
02-11-2011 09:47:44.385 INFO loader - Server supporting SSL v2/v3
02-11-2011 09:47:44.385 INFO loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
And now, with the immutable flag set, I have ssl traffic from forwarder to indexer and log entries show up as expected.
Summary:
If I immute (is that a verb? 🙂 ) the outputs and inputs files, splunk can't muck with them and all works. I just don't want to do this in our production indexer for fear of leaving a little time bomb.
I could swear it wasn't just the passwords when working with it a week ago, but I've slept a bit since then. :)
Thank you so much for offering assistance!
Pete
... View more