I’m setting up a multisite ssl indexing-cluster with splunk 6.4.1 . I want to use indexer discovery.
I succeed in setting up Indexer discovery without ssl.
I succeed in setting up forwarding in ssl without Indexer Discovery. But I always failed using Indexer Discovery and SSL. Here is the situation:
The Search Head tries to forward its data to the “discovered” indexers .
On the master node : (x.x.x.11)
server.conf
[indexer_discovery]
polling_rate = 10
indexerWeightByDiskCapacity = false
pass4SymmKey = mykey
splunkd.log
WARN CMRestIndexerDiscoveryHandler - Forwarder x.x.x.10 and indexer(s) x.x.x.12:9997,x.x.x.13:9997 have mismatching SSL configurations.
On The search head : (x.x.x.10)
splunkd.log
INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=7340032 in bytes.
ERROR TcpOutputProc - target=x.x.x.13:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...
ERROR TcpOutputProc - target=x.x.x.12:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...
outputs.conf
[indexer_discovery:master1]
master_uri = https://x.x.x.11:8089
pass4SymmKey = mykey
[default]
defaultGroup = splunkssl
indexAndForward = false
[tcpout:splunkssl]
sslCertPath = /opt/splunk/etc/mycerts/cert.pem
sslPassword = $1$QqVcFxZnCuOU
sslRootCAPath = /opt/splunk/etc/mycerts/cacert.pem
sslVerifyServerCert = true
forwardedindex.filter.disable = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK = true
On the peers indexers (x.x.x.12,x.x.x.13)
splunkd.log
inputs.conf
[splunktcp-ssl:9997]
disabled = 0
[SSL]
disabled = 0
password = $1$aIEM+AiS56yl
requireClientCert = false
rootCA = /opt/splunk/etc/mycerts/cacert.pem
serverCert = /opt/splunk/etc/mycerts/cert.pem
What do I miss ?
To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :
/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah
To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :
/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah
If using the CLI to configure the receiver (e.g. splunk enable listen 9997), splunk creates the listener under search rather than $SPLUNK_HOME/splunk/etc/system/local.
But, if you follow the instructions in the clustering documentation, you would manually edit inputs.conf in /etc/system/local, instead of using the CLI - so that's probably why splunktcp it was enabled where you weren't expecting.
It doesn't look like it from your example, but do you have more than one input port enabled when you enable discovery?
http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/indexerdiscovery
When using indexer discovery, each peer node can have only a single configured receiving port.