Deployment Architecture

Setting Indexer-discovery in a multi-site cluster-indexer with SSL

breddupuis
Explorer

I’m setting up a multisite ssl indexing-cluster with splunk 6.4.1 . I want to use indexer discovery.
I succeed in setting up Indexer discovery without ssl.
I succeed in setting up forwarding in ssl without Indexer Discovery. But I always failed using Indexer Discovery and SSL. Here is the situation:

The Search Head tries to forward its data to the “discovered” indexers .

On the master node : (x.x.x.11)
server.conf
[indexer_discovery]
polling_rate = 10
indexerWeightByDiskCapacity = false
pass4SymmKey = mykey

splunkd.log
WARN CMRestIndexerDiscoveryHandler - Forwarder x.x.x.10 and indexer(s) x.x.x.12:9997,x.x.x.13:9997 have mismatching SSL configurations.

On The search head : (x.x.x.10)
splunkd.log
INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=7340032 in bytes.
ERROR TcpOutputProc - target=x.x.x.13:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...
ERROR TcpOutputProc - target=x.x.x.12:9997 ssl=0 mismatch with ssl config in outputs.conf for server, skipping...

outputs.conf

[indexer_discovery:master1]
master_uri = https://x.x.x.11:8089
pass4SymmKey = mykey

[default]
defaultGroup = splunkssl
indexAndForward = false

[tcpout:splunkssl]
sslCertPath = /opt/splunk/etc/mycerts/cert.pem
sslPassword = $1$QqVcFxZnCuOU
sslRootCAPath = /opt/splunk/etc/mycerts/cacert.pem
sslVerifyServerCert = true
forwardedindex.filter.disable = true
autoLBFrequency = 30
forceTimebasedAutoLB = true
indexerDiscovery = master1
useACK = true

server = x.x.x.12:9997 <= no ssl issue when uncommented and indexerDiscovery is commented

On the peers indexers (x.x.x.12,x.x.x.13)
splunkd.log
inputs.conf
[splunktcp-ssl:9997]
disabled = 0

[SSL]
disabled = 0
password = $1$aIEM+AiS56yl
requireClientCert = false
rootCA = /opt/splunk/etc/mycerts/cacert.pem
serverCert = /opt/splunk/etc/mycerts/cert.pem

What do I miss ?

1 Solution

breddupuis
Explorer

To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :

/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah

View solution in original post

0 Karma

breddupuis
Explorer

To give a precise answer to Jeremiah, I ran the btool to verify the splunktcp config :
../bin/splunk cmd btool inputs list splunktcp --debug
A surprising part of the result was :

/opt/splunk/etc/apps/search/local/inputs.conf [splunktcp://9997]
I finally found a very misplaced “…/bin/splunk enable listen 9997 “ in my installation procedure.
I still don’t understand why the result is set in Search Apps … where I was not looking at all.
But ... I disabled the splunktcp …/bin/splunk disable listen 9997
and now indexer-discovery and ssl indexing are ok for all the forwarders and even for the master node.
Thanks Jeremiah

0 Karma

gordo32
Communicator

If using the CLI to configure the receiver (e.g. splunk enable listen 9997), splunk creates the listener under search rather than $SPLUNK_HOME/splunk/etc/system/local.

But, if you follow the instructions in the clustering documentation, you would manually edit inputs.conf in /etc/system/local, instead of using the CLI - so that's probably why splunktcp it was enabled where you weren't expecting.

0 Karma

Jeremiah
Motivator

It doesn't look like it from your example, but do you have more than one input port enabled when you enable discovery?

http://docs.splunk.com/Documentation/Splunk/6.4.2/Indexer/indexerdiscovery

When using indexer discovery, each peer node can have only a single configured receiving port.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...