Getting Data In

splunk fowarder SSL error ERROR TcpOutputProc

mcalautti
Explorer

on the forwarder I get
ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server

right before that I see
ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/certs/forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

not sure what that means.
I took the existing certs DIR from an existing splunk build that is working fine with the forwarder

the indexer is version 4.1.5, build 85165
the forwarder is splunkforwarder-4.2.3-105575-Linux-x86_64

output from openssl x509 -inform PEM -in forwarder.pem -text -noout

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
8e:69:04:62:da:36:fa:2b
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=CA, L=SanFrancisco, O=SplunkInc, CN=SplunkCA, O=SplunkUser
Validity
Not Before: Sep 21 18:29:40 2011 GMT
Not After : Sep 20 18:29:40 2014 GMT
Subject: C=US, ST=MA, L=Bedford, O=sb
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:a8:d4:41:84:b2:9f:3c:3f:7f:c3:a2:3d:54:9d:
7f:0b:52:53:73:37:35:85:99:04:ce:09:40:68:38:
5c:ed:0f:52:ff:89:31:e8:b7:c7:f6:82:8d:bc:12:
fa:34:e1:53:65:47:af:4c:5d:03:ab:a0:7f:64:3b:
b2:24:b8:da:db:4e:16:fb:09:3e:11:1f:aa:5e:b3:
b2:20:d6:78:99:3d:ed:c8:74:5b:94:e9:b2:bb:12:
c6:db:85:fa:4c:ec:f3:8b:41:28:6b:03:2e:e4:c6:
11:d4:47:ec:21:c5:8a:70:e0:2f:64:bd:e5:28:f7:
a8:c9:a3:8d:e6:f1:10:b8:59
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
71:91:88:bd:22:cb:aa:45:3c:9e:ec:43:8d:a3:78:cb:d2:f1:
b6:16:a3:66:80:ba:68:55:c8:18:0d:6f:a9:1a:2a:c2:f3:cf:
a0:c2:b6:0a:f1:8b:f4:9b:e1:e4:70:d3:5c:8c:b1:75:2f:bf:
bd:fc:de:e1:bd:c9:a1:ac:54:fe:99:3e:d1:29:9a:a6:9b:b5:
7c:d3:2f:4e:4e:f2:f6:af:a1:0b:cc:e1:d2:e7:1d:3a:27:0c:
7d:21:4e:78:1e:d1:45:32:da:79:0c:b0:8a:48:3f:3f:fa:23:
3e:71:6b:6f:63:c8:21:ee:c8:a6:86:4e:18:b7:40:52:22:29:
48:38

Tags (3)
1 Solution

nrpeter
Engager

This error occurs when splunk tries to decrypt the RSA private key in server.pem with the wrong password.

I'll show you how to fix this by decrypting the cert by hand as long as you are using default splunk signing CAs.

 1. Delete server.pem from your universal forwarder and restart splunk (this will recreate server.pem with a default password of 'password')
 2. Decrypt the private RSA key 'openssl rsa -in server.pem -out server_unprotected.pem'
 3. Since server_unprotected.pem will only have the decrypted RSA private key in it, copy & paste the other two blocks from server.pem (don't copy over the original encrypted RSA private key)
 4. mv splunk_unprotected.pem server.pem
 5. Restart splunk

karthi4k
Explorer

It works like a charm. Thanks peter. One typo in the point 4 where it should be server.pem and not splunk.pem

0 Karma

wqiu_splunk
Splunk Employee
Splunk Employee

this should be the answer indeed!!!!

0 Karma

dhillny
Engager

life saver bro, other than the typo on #4. This is a solid fix.

0 Karma

moseisleydk
Path Finder

We just ran into this using conf files working fine on a OpenSolaris 32-bit - not working on Ubuntu 10.04 TLS (64-bit)

The problem was the decryption of the password, pasting the real in as plain text worked, and the resulting encrypted string was different on the Ubuntu.

andyspusm
Explorer

I just saw the same behavior between two Windows 7 Professional x64 machines - pasting the password in plaintext worked. Thanks for the idea.

0 Karma

mikelanghorst
Motivator

I'd start with looking at the forwarder.pem file with vi and openssl first.

With vi, it should be readable text with "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" blocks. Next use openssl and inspect the data returned:
openssl x509 -inform PEM -in -text -noout

0 Karma

mcalautti
Explorer

ok.. I see the CERT in the file..

i added the output from openssl to the description.. looks ok to me.. what do you think ?
thanks
mike

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...