Getting Data In

Why do I get this error when configuring the universal forwarder: SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed?

wouterr
Explorer

Hi,

I am installing the universal forwarder (6.2) on redhat. I am running into several issues with the SSL setup. I am using my own selfsigned certs. This is working fine in an old 4.2 universal forwarder setup.

After extracting splunk I do the following:

1) Copy my certs to /etc/auth/server.pem and /etc/auth/ca.pem

2) update /etc/system/local/inputs.conf with

...
[tcpout-server://splunkserver.ec2.local:9997]
sslCertPath = /usr/local/splunkforwarder/etc/auth/server.pem
sslPassword = mypassword
sslRootCAPath = /usr/local/splunkforwarder/etc/auth/ca.pem
sslVerifyServerCert = false
...    

3) Update etc/system/default/server.conf

...
sslPassword = mypassword
...

4) Start splunk server with no configuration errors and etc/system/local/server.conf is generated

5) Find this error in splunkd.log

08-04-2016 13:07:13.134 -0700 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

What am I missing? Do I need to care about splunk.secret that manages the encryption of the sslpassword value in /etc/sytem/local/outputs and /etc/system/local/server.conf?

I can open my cert with password when doing:

openssl rsa -in /usr/local/splunkforwarder/etc/auth/server.pem -text

So the cert and passphrase is correct. What else should I consider? I have stopped splunk and set the sslKeysfilePassword in etc/system/local/server.conf. Start splunk but no luck. I have also tried the same for the sslPassword in etc/system/local/outputs.conf but not luck.

Any advice would be appreciated

Thanks,
Wouter

dwaddle
SplunkTrust
SplunkTrust

A leadoff comment - do not make changes to things in $SPLUNK_HOME/etc/system/default -- or in any other app-level default unless you are the author of the app. Changes made in default will be overwritten without warning during upgrades, and making the right change in a local file will override the defaults anyway.

Splunk uses different SSL settings in different configuration files for each type of connection. I'm not sure if your notes above are correct or not, because you've got a [tcpout-server] stanza listed in inputs.conf (it should be in outputs.conf), and you're updating a password in etc/system/default/server.conf that is most undoubtedly being overlaid by etc/system/local/server.conf.

By self-signed do you mean actually self-signed, or do you mean signed by your private certificate authority? These are two different things. Self-signed is like a tautology -- I am who I say I am because I say that is who I am. A private CA has a root certificate (which is probably self-signed) that establishes a trust anchor for other certificates that the CA signs.

I would use btool to look at how the various SSL settings are configured. Something like:

splunk cmd btool --debug outputs list

Might help you discover a configuration file somewhere that does have a setting like sslVerifyServerCert = true, which is what the original error you reported sounds like. Situational awareness of where settings are done, and how different configuration files overlay, is of absolute importance when trying to understand why Splunk is doing something. The docs talk about this at http://docs.splunk.com/Documentation/Splunk/6.4.2/Admin/Wheretofindtheconfigurationfiles

Finally, (plugging mine and @starcher's work some) you should have a look at this .conf talk material:

http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4
http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPra...

This is a whole cookbook intended to help you configure all the SSL things in Splunk correctly.

wouterr
Explorer

Hi Stephen,

I did not solve this problem. Instead of attempting to fix this we decided to blow away the old install and re-install from scratch. That seemed to have resolved this issue for some reason. Sorry not very helpful if you are in a situation where you cant afford to do that.

Cheers
Wouter

0 Karma

stepheneardley
Explorer

That's unfortunate but thanks for letting me know. A reinstall really isn't possible at this point. I'll keep digging. I'll update you if I do find a solution though.

0 Karma

wouterr
Explorer

Hi

You are correct I mistakenly listed inputs.conf instead of outputs.conf in step 2 in the initial question.

I made a change to $SPLUNK_HOME/etc/system/default/server.conf knowing that it is probably not a good idea but how else can I change the default sslPassword? Seems to be propagated when I start the server to $SPLUNK_HOME/etc/system/local/server.conf (which is not there on first startup). I should probably start the app up the first time then ensure I overwrite the correct sslpassword to $SPLUNK_HOME/etc/system/local/server.conf after?

By self signed I mean that I created it by following the splunk community wiki instructions for self signed cert with new root CA. http://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

I used the splunk btool before to check configuration values but did not explicitly look for sslVerifyServerCert = true. Thanks for the tip, I will look out for that.

Thanks for the links. I will educate myself before I dig into the problem again

0 Karma

stepheneardley
Explorer

Did you ever figure this out wouterr? I'm hitting the same problem in 6.4

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...