All Apps and Add-ons

Splunk 6.6 upgrade seems to have permissions issues

earlhelms
Path Finder

After doing an rpm upgrade to 6.6 I'm having some pretty big issues that appear to be permission related.
Examples:
Failed to start KV Store process. See mongod.log and splunkd.log for details.

When I tried to search the splunkd log I received more errors...
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:config' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:hipmatch' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:system' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:threat' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified destination fields in the lookup table.' for conf 'pan:traffic' and lookup table 'pan_vendor_info_lookup'.
•Error 'Could not find all of the specified lookup fields in the lookup table.' for conf 'pan:threat' and lookup table 'threat_lookup'.

Fortunately, this is on my dev box.

0 Karma
1 Solution

earlhelms
Path Finder

I resolved my own issue...
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server -c zzzz-zzzz.zzz.zzzz.com -l 2048
Note: I edited the FQDN in the above example

View solution in original post

earlhelms
Path Finder

I resolved my own issue...
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server -c zzzz-zzzz.zzz.zzzz.com -l 2048
Note: I edited the FQDN in the above example

damode
Motivator

This will just recreate the default cert. In my case, we are using our own cert. Still getting the same error.
I checked file permission issue on Splunk.key file and that also seems fine.

0 Karma

earlhelms
Path Finder

A related link: https://answers.splunk.com/answers/457893/after-upgrading-to-650-kv-store-will-not-start.html

Which led me to this...
[root@zzz]# tail /opt/splunk/var/log/splunk/mongod.log
2017-05-08T14:35:11.400Z W CONTROL No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
2017-05-08T14:35:11.416Z F NETWORK The provided SSL certificate is expired or not yet valid.
2017-05-08T14:35:11.416Z I - Fatal Assertion 28652
2017-05-08T14:35:11.416Z I -
***aborting after fassert() failure

and this...
[root@zzzzz]# openssl x509 -enddate -noout -in ./server.pem
notAfter=Apr 13 20:57:57 2017 GMT

Still not entirely sure what to do

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...

Updated Data Management and AWS GDI Inventory in Splunk Observability

We’re making some changes to Data Management and Infrastructure Inventory for AWS. The Data Management page, ...