Deployment Architecture

Setting Indexer-discovery in a multi-site cluster-indexer with SSL

breddupuis
Explorer

I’m setting up a multisite ssl indexing-cluster with splunk 6.4.1 . I want to use indexer discovery.
I succeed in setting up Indexer discovery without ssl.
I succeed in setting up forwarding in ssl without Indexer Discovery. But I always failed using Indexer Discovery and SSL. Here is the situation:

The Search Head tries to forward its data to the “discovered” indexers .

On the master node : (x.x.x.11)
server.conf
[indexer_discovery]
polling_rate = 10
indexerWeightByDiskCapacity = false
pass4SymmKey = mykey

splunkd.log
WARN CMRestIndexerDiscoveryHandler - Forwarder x.x.x.10 and indexer(s) x.x.x.12:9997,x.x.x.13:9997 have mismatching SSL configurations.

On The search head : (x.x.x.10)
splunkd.log
INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=7340032 in bytes.
ERROR TcpOutputProc - target=x.x.x.13:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...
ERROR TcpOutputProc - target=x.x.x.12:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...

outputs.conf

[indexer_discovery:master1]
master_uri = https://x.x.x.11:8089
pass4SymmKey = mykey

[default]
defaultGroup = splunkssl
indexAndForward = false

[tcpout:splunkssl]
sslCertPath = /opt/splunk/etc/mycerts/cert.pem
sslPassword = $1$QqVcFxZnCuOU
sslRootCAPath = /opt/splunk/etc/mycerts/cacert.pem
sslVerifyServerCert = true
forwardedindex.filter.disable = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK = true

server = x.x.x.12:9997 <= no ssl issue when uncommented and indexerDiscovery is commented

On the peers indexers (x.x.x.12,x.x.x.13)
splunkd.log
inputs.conf
[splunktcp-ssl:9997]
disabled = 0

[SSL]
disabled = 0
password = $1$aIEM+AiS56yl
requireClientCert = false
rootCA = /opt/splunk/etc/mycerts/cacert.pem
serverCert = /opt/splunk/etc/mycerts/cert.pem

What do I miss ?

1 Solution

breddupuis
Explorer

To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :

/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah

View solution in original post

0 Karma

breddupuis
Explorer

To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :

/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah

0 Karma

gordo32
Communicator

If using the CLI to configure the receiver (e.g. splunk enable listen 9997), splunk creates the listener under search rather than $SPLUNK_HOME/splunk/etc/system/local.

But, if you follow the instructions in the clustering documentation, you would manually edit inputs.conf in /etc/system/local, instead of using the CLI - so that's probably why splunktcp it was enabled where you weren't expecting.

0 Karma

Jeremiah
Motivator

It doesn't look like it from your example, but do you have more than one input port enabled when you enable discovery?

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/indexerdiscovery

When using indexer discovery, each peer node can have only a single configured receiving port.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...