Knowledge Management

KV Store lookup failing with error about KV store initialization failure

nnmiller
Contributor

KV store lookups are failing with the following error:

Error in 'inputlookup' command: External command based lookup 'kvstore_lookup' is not available because KV Store initialization has failed. 

Verified no orphaned mongod.lock file or firewall port issues per Splunk Answer 208349. Attempted manual restart of mongod with

./mongod --fork --logpath $SPLUNK_HOME/var/log/splunk/test_mongodb.log
about to fork child process, waiting until server is ready for connections.
forked process: 4234
ERROR: child process failed, exited with error number 100

Searching for above error, noted a post on StackOverflow where permissions were a problem, so I verified permissions. Mongod still did not start.

1 Solution

nnmiller
Contributor

Closer review of mongod.log showed the following errors:

mongod.log: 2016-04-27T16:42:40.111Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
mongod.log: 2016-04-27T16:42:40.129Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

First line appears to be spurious, second log entry was the problem. Splunk was looking at /opt/splunk/etc/auth/server.pem, showing in successful, earlier starts of mongod. $SPLUNK_HOME/etc/auth/server.pem is the default path and cert. Verified expiry by running: $SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

Replaced with a newly generated cert, kv store started per norm.

In recent versions of Splunk you can regenerate the server's certificate using the following steps:

  • Run the following command to update the server certificate:

    /opt/splunk/bin/splunk createssl server-cert -d <path_to_rootCA> -n server.pem -c <dns_name_of_host> -p
    

    The default location for the rootCA files is $SPLUNK_HOME/etc/auth/.

  • If you know the current sslKeysfilePassword for the search head, you can use the same password when creating the new certificate and won't need to change anything else. If you do not know the search head's plaintext sslKeysfilePassword then you will need to edit $SPLUNK_HOME/etc/system/local/server.conf and update the sslKeysfilePassword in the [sslConfig] stanza to match the new passphrase you used when generating the new server cert.

  • In older versions of Splunk, you can use the script $SPLUNK_HOME/bin/genSignedServerCert.sh

View solution in original post

nnmiller
Contributor

Closer review of mongod.log showed the following errors:

mongod.log: 2016-04-27T16:42:40.111Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
mongod.log: 2016-04-27T16:42:40.129Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2

First line appears to be spurious, second log entry was the problem. Splunk was looking at /opt/splunk/etc/auth/server.pem, showing in successful, earlier starts of mongod. $SPLUNK_HOME/etc/auth/server.pem is the default path and cert. Verified expiry by running: $SPLUNK_HOME/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem

Replaced with a newly generated cert, kv store started per norm.

In recent versions of Splunk you can regenerate the server's certificate using the following steps:

  • Run the following command to update the server certificate:

    /opt/splunk/bin/splunk createssl server-cert -d <path_to_rootCA> -n server.pem -c <dns_name_of_host> -p
    

    The default location for the rootCA files is $SPLUNK_HOME/etc/auth/.

  • If you know the current sslKeysfilePassword for the search head, you can use the same password when creating the new certificate and won't need to change anything else. If you do not know the search head's plaintext sslKeysfilePassword then you will need to edit $SPLUNK_HOME/etc/system/local/server.conf and update the sslKeysfilePassword in the [sslConfig] stanza to match the new passphrase you used when generating the new server cert.

  • In older versions of Splunk, you can use the script $SPLUNK_HOME/bin/genSignedServerCert.sh

briancronrath
Contributor

You are a life saver

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

In newer versions of Splunk, sslKeysfilePassword is deprecated in favor of sslPassword. See https://docs.splunk.com/Documentation/Splunk/7.2.4/Admin/Serverconf for more info about these settings.

0 Karma

varunCarbyne
Explorer

We faced the SSL problem and when we try to check the certificate validity, we see

notAfter= some date of 2020 GMT

Please suggest any other option for same

0 Karma

Shephali
Explorer

Hi @varunCarbyne ,

Did you figure out the solution?
I have a similar issue at my end.SSL Certificate validity is fine,still KV initialization failing

0 Karma

Waltersr24
New Member

where would you go to generate the new cert? im new with Splunk

0 Karma

ykpramodh
Engager

Hi,

We faced the kv store problem where kv store stays in "starting" status. Checked the mongod.log and identified the SSL problem suggested above.

We faced the SSL problem and when we try to check the certificate validity, we see

notAfter=Dec 9 19:01:45 2019 GMT

Can you please suggest any other options we should try?

nnmiller
Contributor

If KV Store stays in "Starting" status, then you have a different problem, assuming mongod is actually running. ERROR: child process failed, exited with error number 100 is a generic error.

I would suggest opening a support ticket if you haven't resolved the problem already.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...