Setting Indexer-discovery in a multi-site cluster-indexer with SSL

[breddupuis]

I’m setting up a multisite ssl indexing-cluster with splunk 6.4.1 . I want to use indexer discovery.
I succeed in setting up Indexer discovery without ssl.
I succeed in setting up forwarding in ssl without Indexer Discovery. But I always failed using Indexer Discovery and SSL. Here is the situation:

The Search Head tries to forward its data to the “discovered” indexers .

On the master node : (x.x.x.11)
server.conf
[indexer_discovery]
polling_rate = 10
indexerWeightByDiskCapacity = false
pass4SymmKey = mykey

splunkd.log
WARN CMRestIndexerDiscoveryHandler - Forwarder x.x.x.10 and indexer(s) x.x.x.12:9997,x.x.x.13:9997 have mismatching SSL configurations.

On The search head : (x.x.x.10)
splunkd.log
INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=7340032 in bytes.
ERROR TcpOutputProc - target=x.x.x.13:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...
ERROR TcpOutputProc - target=x.x.x.12:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...

outputs.conf

[indexer_discovery:master1]
master_uri = https://x.x.x.11:8089
pass4SymmKey = mykey

[default]
defaultGroup = splunkssl
indexAndForward = false

[tcpout:splunkssl]
sslCertPath = /opt/splunk/etc/mycerts/cert.pem
sslPassword = $1$QqVcFxZnCuOU
sslRootCAPath = /opt/splunk/etc/mycerts/cacert.pem
sslVerifyServerCert = true
forwardedindex.filter.disable = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK = true

server = x.x.x.12:9997 <= no ssl issue when uncommented and indexerDiscovery is commented

On the peers indexers (x.x.x.12,x.x.x.13)
splunkd.log
inputs.conf
[splunktcp-ssl:9997]
disabled = 0

[SSL]
disabled = 0
password = $1$aIEM+AiS56yl
requireClientCert = false
rootCA = /opt/splunk/etc/mycerts/cacert.pem
serverCert = /opt/splunk/etc/mycerts/cert.pem

What do I miss ?

[breddupuis · Accepted solution]

To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :

/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah

[gordo32]

If using the CLI to configure the receiver (e.g. splunk enable listen 9997), splunk creates the listener under search rather than $SPLUNK_HOME/splunk/etc/system/local.

But, if you follow the instructions in the clustering documentation, you would manually edit inputs.conf in /etc/system/local, instead of using the CLI - so that's probably why splunktcp it was enabled where you weren't expecting.

[Jeremiah]

It doesn't look like it from your example, but do you have more than one input port enabled when you enable discovery?

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/indexerdiscovery

When using indexer discovery, each peer node can have only a single configured receiving port.