External command based lookup 'tSessions' is not available because KV Store initialization has failed.

[banderson7]

Search head cluster running 6.3 and Splunk App for Windows Infrastructure 1.20. I'm getting these errors for my scheduled searches:

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tSessions_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tSessions' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;tHostInfo_Lookup_Update", message="Error in 'inputlookup' command: External command based lookup 'tHostInfo' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed

ERROR SavedSplunker - savedsearch_id="nobody;splunk_app_windows_infrastructure;DomainSelector_Lookup", message="Error in 'outputlookup' command: External command based lookup 'DomainSelector' is not available because KV Store initialization has failed. Please contact your system administrator.". No actions executed
On all three search heads, the permissions for /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key is

-rw------- 1 splunk splunk 88 Oct 22 11:42 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

and these errors occur even after splunk is restarted in the environment, so I think that rules out a mongod restart.
This stops the majority of the splunk app for windows infrastructure from displaying results.

[daymauler]

In our case, the issue was as a result of the expired certs:

Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator

Here is the fix:

1. splunkd.log: ERROR KVStoreLookup - External command based lookup 'app_name' is not available because KV Store initialization has not completed yet......
2. mongod.log: The provided SSL certificate is expired or not yet valid....
3. Check Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Sep 8 17:56:51 2019 GMT
4. Stopped Splunk service
5. Renamed current server.pem: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem
6. Restarted Splunk services
7. Checked Certs: /opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem notAfter=Oct 24 17:38:28 2022 GMT
8. Results where displayed for app.

[daymauler]

Step 5 is Renamed current server.pem to server.pem.old: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem.old

[woodcock]

In my case it was that the certificates and other files were too permissive. I did this to fix it:

cd /opt/splunk/var/lib/splunk/kvstore/mongo
chmod 600 *
/opt/splunk/bin/splunk restart

[nnmiller]

Have a look at these two Answer posts:
Mongod fails to start due to SSL cert expiration
Mongod fails to start

It may also be a permissions problem in the mongod tree, so verify that as well.