Security

Event not detected by indexer on [splunktcp-ssl] port

oluwoleoyetoke
Explorer

I configured splunk to ingest logs on port 9338 with SSL enabled.

TCP dump on the port shows log data being received, but when I search on the indexer, this event is not captured

sudo tcpdump -i any port 9338

21 packets captured
42 packets received by filter
0 packets dropped by kernel

the log file in my /opt/splunk/var/log/splunk/splunkd.log

01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:37.519 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 06:26:37.520 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 06:26:38.343 +0000 WARN  HttpListener - Socket error from 127.0.0.1:44420 while idling: error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request

Please how can I fix this?

0 Karma
1 Solution

oluwoleoyetoke
Explorer

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

View solution in original post

0 Karma

oluwoleoyetoke
Explorer

This was solved by changing the

 [splunktcp-ssl://:9338]
 disabled = 0

to

 [tcp-ssl://:9338]
 disabled = 0

splunktcp-ssl should be used if the forwarder is a Splunk forwarder. But in my case I was using Splunk's Java logging library to forwards logs to the Splunk instance.

I have added a gist here (https://gist.github.com/OluwoleOyetoke/b38aecbb47323ad25840d711cef8bf1) to help others who may need to do the same in the future

0 Karma

oluwoleoyetoke
Explorer

Additional Log

01-23-2020 07:35:13.079 +0000 INFO  IndexWriter - openDatabases complete currentId=-1 idx=summary
01-23-2020 07:35:13.079 +0000 INFO  IndexProcessor - Initializing indexes took usec=700 reloading=false indexes_initialized=9
01-23-2020 07:35:13.087 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9336 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9337 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for raw input
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - Creating FwdDataSSLConfig SSL context. Will open port=IPv4 port 9338 with compression=1
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 is reserved for splunk 2 splunk (SSL)
01-23-2020 07:35:13.088 +0000 INFO  TcpInputConfig - IPv4 port 9338 will negotiate s2s protocol level 6
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9336 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 9337 with Non-SSL
01-23-2020 07:35:13.088 +0000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9338 with SSL
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor -     interval: 0 ms
01-23-2020 07:35:13.090 +0000 INFO  ExecProcessor - interval="0 * * * *" is a valid cron schedule
0 Karma

oluwoleoyetoke
Explorer

Input.conf

[splunktcp-ssl://:9338]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/mycerts/myNewServerCertificate.pem
sslPassword = $7$0XwJ4Q3QxJXxrDeuKBzS3XYyHaNhoBc05xBbcnkb1miQVcrEFlEfEixMjTv3arI99g==
requireClientCert = false
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Have you configured intermediateCA or rootCA in server.conf on Indexer ? (Ref doc. https://docs.splunk.com/Documentation/Splunk/8.0.1/Security/ConfigureSplunkforwardingtousesignedcert...)

0 Karma

oluwoleoyetoke
Explorer

No, I have not done this yet. I will do it and update this thread if it solves the problem. Thanks

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...