SSL connection between Indexer and Forwarder

[garima_chauhan]

Hi,

I am not able to configure the ssl connections between the forwarder and indexer. The splunkd logs on both the indexer and forwarder are not the same as cited in the documentation.

Here is what I get on Indexer in splunkd.log:
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is not compressed
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
Date Time +1100 INFO TcpInputConfig - IPv4 port 9997 is compressed
Date Time +1100 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

After this, I do not get any other message as mentioned in the documentation.

On Forwarder, I get the following in splunkd.log:
Date Time +1100 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
Date Time +1100 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
Date Time +1100 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
Date Time +1100 INFO TcpOutputProc - tcpout group splunkssl using Auto load balanced forwarding
Date Time +1100 INFO TcpOutputProc - Group splunkssl initialized with maxQueueSize=512000 in bytes.

Date Time +1100 WARN TcpOutputProc - Connected to idx=:9997. Not using ACK.

Date Time +1100 WARN TcpOutputProc - Connected to idx=:9997. Not using ACK.

Date Time +1100 INFO TcpOutputProc - Connection to :9997 closed. Connection closed

For enabling SSL connection between a forwarder and an indexer, I performed the following configurations:

On Indexer(Windows)
I added the following stanzas in $SPLUNK_HOME\etc\system\local\inputs.conf

[splunktcp-ssl:9997]
compressed = true

[SSL]
rootCA = $SPLUNK_HOME\etc\auth\cacert.pem
serverCert = $SPLUNK_HOME\etc\auth\server.pem
password = password

On Forwarder(Windows)
I added the following stanzas in $SPLUNK_HOME\etc\system\local\outputs.conf

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = :9997
sslCertPath = $SPLUNK_HOME\etc\auth\server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME\etc\auth\cacert.pem
sslVerifyServerCert = false

I am not able to figure out where am I making a mistake.Please help.

[jtacy]

The log on the indexer suggests that you might have two input.conf stanzas using port 9997. Can you verify that you don't have a regular splunktcp:9997 stanza in there? If you want to support both non-SSL and SSL forwarders, you'll need to choose a different port for either of the inputs. Here's working configuration from a forwarder and indexer:

Indexer

[splunktcp-ssl:9997]

[SSL]
password = password
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

Forwarder

[tcpout]
defaultGroup = indexer01

[tcpout:indexer01]
server = indexer01:9997
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = password
sslRootCAPath = $SPLUNK_HOME/etc/auth/ca.pem

I'm not using compression in my example since the documentation for inputs.conf says that it only applies to non-SSL inputs. I'd remove that from the indexer and forwarder config and restart both instances to make sure that's not part of the problem, but my main concern is the possibility that there are two listeners using the same port.

With the above configuration, you should start seeing events for your forwarder host in the _internal index almost right away. Good luck!

[garima_chauhan]

Thanks jtacy. I am able to see the metrics.log as you mentioned.

[jtacy]

Couple of links about using custom certs:
http://docs.splunk.com/Documentation/Splunk/latest/Security/Howtoself-signcertificates
http://docs.splunk.com/Documentation/Splunk/latest/Security/ConfigureSplunkforwardingtousesignedcert...

[jtacy]

If you want to be really sure, you could use Wireshark to capture traffic between the forwarder and indexer on your SSL port. Capture for a couple of minutes to make sure you see several connections (assuming 30s connect interval). Follow one of the TCP streams and the only text that should be readable includes things like SplunkCommonCA that pertain to the default certs.

This, combined with the info from metrics.log on the indexer, should give you pretty good confidence that SSL is working properly. If you're that concerned about SSL, make sure you're using a custom CA and verifying certs.

[jtacy]

Here's a good entry from metrics.log on an indexer. Note connectionType=cookedSSL and ssl=true:

11-21-2013 18:21:04.211 -0600 INFO Metrics - group=tcpin_connections, 10.10.10.10:57308:9997, connectionType=cookedSSL, sourcePort=57308, sourceHost=10.10.10.10, sourceIp=10.10.10.10, destPort=9997, _tcp_Bps=157.95, _tcp_KBps=0.15, _tcp_avg_thruput=0.15, kb=2.31, _tcp_Kprocessed=2.31, _tcp_eps=0.27, build=163460, version=5.0.3, os=Windows, arch=x64, hostname=forwarder, guid=ABCDEF12-1234-ABCD-1234-ABCDEF123456, fwdType=uf, ssl=true, lastIndexer=None, ack=false

[garima_chauhan]

Hi, I have no duplicate inputs.conf. In splunkd.log, it still doesn't show me the stated output, however,in metrics.log, it is giving that ssl=true.Would this mean, that ssl is enabled now?

Please help.

[jtacy]

Unless the "timed out" message is appearing repeatedly in the logs, if the events are being indexed you might be OK. However, again, I was only able to match your indexer log output when I intentionally added duplicate inputs. My live indexers output only one set of TcpInputConfig events when starting:

10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
10-29-2013 06:09:51.078 -0500 INFO TcpInputConfig - IPv4 port 9997 is not compressed

Have you run "splunk btool inputs list" just to double-check that you don't have a duplicate?

[garima_chauhan]

Hi jtacy,
No I dont have a regular splunktcp-9997 stanza already present in the inputs.conf. Still, I created and used a separate port-9996 and changed the config files as given by you. I am still not getting the desired output. While the logs on indexer are same, the forwarder splunkd logs also give the following line:

Date Time +1100 WARN TcpOutputProc - Cooked connection to ip=IndexerIP:9996 timed out

The data from forwarder is successfully being indexed in the indexer, though. Please suggest what else can be the problem here.