KVStore Initialization

[jsmith_splunk]

I'm trying to install Enterprise Security 4 on Splunk 6.3 and it is hanging on the installing apps phase. I've restarted this about 6 times with no effect. I've verified the file permissions and ownership (chmod 755, and chown splunk.splunk. Splunkd is running as the user splunk.

In the splunkd.log on this server I see thousands of errors regarding failure to initialize the kvstore. A few examples are:

-Error in 'outputlookup' command: External commands based lookup 'action_history_lookup' is not available because KV Store initialization has failed.

-ERROR KVStoreLookup -failed to create output lookup context

-Failed to execute KV Store lookups: External command based lookup 'action_history_lookup' is not available because KV Store initialization has failed.

Does anyone have any insight into this? There are 6 servers in this environment (2 SH, 2 Ind, 1 CM, 1 DS) at the moment and the server I'm trying to place ES on is the only one generating these errors.

[dmuraleetcs]

Thanks, Below command fix that issue.

chmod 400 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

I was not able to open Splunk Add on builder TA due to this issue. Now its been fixed by changing the permission to the mongo key file.

[fairje]

mogodb does not like excessive permissions. If you did a 755 to your entire installation folder I suggest you retract this back.

chmod o-rwx /opt/splunk -R

Assuming splunk is installed in /opt/splunk

This should fix your KVStore issues. Not sure if it will fix your overall issues though, post back your findings after fixing mongodb.

[jsmith_splunk]

Thank you for feedback, typically I would use chmod o-rwx but the admin on this site asked me to use 755 instead, however since I had nothing to lose I changed it to o-rwx.

This doesn't appear to have had any effect, I restarted splunk and restarted the ES installation. ES is still hanging on the installing applications piece and the same KV Store errors I listed above have appeared in splunkd.log since the restart.

[fairje]

Sorry, correction on this, mongodb should have a 400 permission. I would specifically fix this:

chmod 400 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

See if that clears the error.

I would also check out the following Answer as it might help pinpoint if this is indeed your issue, since it is also suggested in there that an upgrade to 6.3 didn't refresh one user's SSL certificates which also was causing an issue with the KVStore:

https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html

[landen99]

400 cleared my kvStoreStatus error. Thank you.

curl -k -s https://localhost:8089/services/server/info | grep kvStore

[fairje]

But yes, I had to have a conversation with our Linux engineer that excessive permissions is not the way to go. It is general all around bad security practices anyway. Depending on the security requirements in your organization this might even be a policy violation since you would have effectively set global read/execute permissions to your collected log data (which is bad). That was why I suggested raking back the global permissions across your install.

Linux, unlike Windows, will get quite upset with you for doing bad permissions.