In our case, the issue was as a result of the expired certs:
Error: Error in 'inputlookup' command: External command based lookup 'app_name' is not available because KV Store initialization has failed. Contact your system administrator
Here is the fix:
splunkd.log: ERROR KVStoreLookup - External command based lookup 'app_name' is not available because KV Store initialization has not completed yet......
mongod.log: The provided SSL certificate is expired or not yet valid....
Check Certs:
/opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
notAfter=Sep 8 17:56:51 2019 GMT
Stopped Splunk service
Renamed current server.pem: mv /opt/splunk/etc/auth/server.pem /opt/splunk/etc/auth/server.pem
Restarted Splunk services
Checked Certs:
/opt/splunk/bin/splunk cmd openssl x509 -enddate -noout -in /opt/splunk/etc/auth/server.pem
notAfter=Oct 24 17:38:28 2022 GMT
Results where displayed for app.
... View more