Why am I seeing TcpInputFd SSL Errors every 10 minutes on all of splunk instances?

[Lowell]

I've been seeing a bunch of error message sequences like this:

01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error = error:00000000:lib(0):func(0):reason(0)
01-07-2011 09:45:40.106 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
01-07-2011 09:45:40.106 ERROR TcpInputFd - SSL Error for fd from HOST:splunk.my.domain, IP:192.168.1.99, PORT:3959

This is happening not only on my splunk indexer, but also on most (but not all) of my splunk forwarders too.

Observations:

• The "HOST" and "IP" are always pointing to my central splunk indexes.
• It seems like errno:0 could indicate that there is NO error (since low-level OS calls often use a return code 0 to mean "success", and any other value indicates a specific error code.)
• Sometime I see SSL_ERROR_SYSCALL ret errno:104 instead of errono:0 but that occurs much less frequently (and the subsequent errors show above are the same)

Any ideas? Should I be concerned about these?

[bohrasaurabh]

In our environment we saw a lot of ERROR TcpInputFd - SSL_ERROR_SYSCALL ret errno:104 on our Deployment server for the forwarders sitting behind DMZ. We requested our security team to open the firewall and the error messages disappeared.

[Lowell · Accepted solution]

After some network packet tracking with wireshark and a bunch of head scratching; I've finally found the answer:

The messages are a result of the "Splunk Monitoring" app that I installed some time ago and had forgotten about. The app is setup to poll a given list of splunk servers (which was out of date on my system, hence not ALL of my splunk instances were reporting this.) The app uses a scripted input that runs on a regular interval (10 minutes on my system.) The basic ping-like approach used by the app actually uses nmap to look for a response on port 8089 (which is the the splunkd HTTPS listener). So because there is no actual request being made, nothing gets logged in the splunkd_access log, and because there is no attempt to handle the SSL stuff, you end up with errors coming from splunkd about an invalid connection. Which is accurate.

So the bottom line is that these messages are not harmful, but there may be a better way to monitor for up/down activity that using nmap. Perhaps there is some internal "ping" like URL that could be used with wget or curl or something like that.

---

Since this took a while track down, I figure it was worth sharing. And of course any other kind of port scanning device or simple TCP monitoring service would most likely also trigger this kind of messages too. Perhaps this will save somebody some time.

[swhiting]

I know this thread is ancient, but wanted to ask Jason if he ever figured out if the ssl errors related to his deployment server issues?

[Jason]

I am also seeing these errors, identical except for errno is errno: 104 - and we're seeing horrible performance in Deployment Server. We're still investigating.