All Apps and Add-ons

Splunk App for Windows Infrastructure: How to troubleshoot error "Key value store must be enabled. Please enable it."?

jawebb
Explorer

Hello all,

I have searched through much of Splunk Answers and Google and have yet to find a working solution for me on this issue. I have Splunk 6.3 and am trying to set up the Splunk App for Windows Infrastructure and have been unable to get it working properly due to kvstore issues. I have seen other threads and have tried all of the proposed solutions.

I believe it to be a permissions issue, but all of the solutions are for systems on Linux boxes. We have ours on a Windows box, so I'm not sure exactly what the proper permissions should be.

alt text

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

These may be the threads you are referring to, but in case not, I will put them there for others that are coming to this page for answers:

https://answers.splunk.com/answers/268584/splunk-app-for-windows-infrastructure-it-has-error.html
https://answers.splunk.com/answers/203979/splunk-app-for-microsoft-exchange-how-to-get-the-t.html
https://answers.splunk.com/answers/206030/splunk-app-for-windows-infrastructure-why-am-i-get-1.html

Have you found any errors or issues in $SPLUNK_HOME/var/log/splunk/mongod.log?
You already tried getting rid of mongo.lock ?

What makes you think it is a permissions issue ?

0 Karma

sobrien
Splunk Employee
Splunk Employee

Hi Jawebb,

You'll need to ensure that the owner of the Splunk folder structure is the same user that is used to run Splunk. This could be the system account, or a domain account. This ownership needs to be set, along with the permissions. Once this is done, restart Splunk.

Hope this helps. Please let me know if it does, and I'll submit this as an answer.

0 Karma

jawebb
Explorer

Here is the log - Basically it's only cert errors from the bottom until this date. We started receiving those once we switched to a third-party cert.

2015-10-12T13:05:02.793Z [conn1315] dropDatabase s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ finished
 2015-10-12T13:05:02.793Z [conn1315] allocating new ns file D:\Program Files\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.ns, filling with zeroes...
 2015-10-12T13:05:02.824Z [FileAllocator] allocating new datafile D:\Program Files\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.0, filling with zeroes...
 2015-10-12T13:05:02.855Z [FileAllocator] done allocating datafile D:\Program Files\Splunk\var\lib\splunk/kvstore\mongo\s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.0, size: 16MB,  took 0.036 secs
 2015-10-12T13:05:02.855Z [conn1315] build index on: s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.c properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.c" }
 2015-10-12T13:05:02.855Z [conn1315]     added index to empty collection
 2015-10-12T13:05:02.855Z [conn1315] build index on: s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.c properties: { v: 1, unique: true, key: { _user: 1, _key: 1 }, name: "_UserAndKeyUniqueIndex", ns: "s_splunkiTR2RCAYp7Go4kZlq1TnMAm9_tHostIOJejUoOGxdcrt2VUpbxhK3iJ.c", background: true }
 2015-10-12T13:05:02.855Z [conn1315]     added index to empty collection
 2015-10-12T13:05:02.902Z [conn1315] end connection 127.0.0.1:51492 (4 connections now open)
 2015-10-12T13:09:34.947Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
 2015-10-12T13:09:34.947Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
 2015-10-12T13:09:35.212Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
 2015-10-12T13:16:09.070Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
 2015-10-12T13:16:09.086Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
 2015-10-12T13:16:09.491Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
 2015-10-12T13:39:40.881Z W CONTROL  No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
 2015-10-12T13:39:40.897Z I CONTROL  Hotfix KB2731284 or later update is not installed, will zero-out data files
 2015-10-12T13:39:41.365Z I CONTROL  dbexit: The provided SSL certificate is expired or not yet valid. rc: 2
0 Karma

jkat54
SplunkTrust
SplunkTrust

try ./splunk cmd btool web list --debug | grep caCertPath see the file it points to, update that path in that config file OR insure the appropriate ca and private key paths are good:

http://docs.splunk.com/Documentation/Splunk/6.3.2/Admin/Webconf#web.conf.example

0 Karma

jkat54
SplunkTrust
SplunkTrust

ensure the permissions are correct on the certs as well.

0 Karma

jawebb
Explorer

Permissions are good and the file listed from the command points to the new certificates

0 Karma

jawebb
Explorer

Thanks for the info. Unfortunately, that didn't seem to do the trick. It was set to local admins for the folder structure, which the user was part of. I changed ownership to be the user itself, restarted, and still receive the error.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...