OK,
We are standing up several new instances of Splunk 7.x and my Dev instance needed certs installed. I did this the other day and afterward the web service would not start. This was because some of the cert files were incorrectly placed. We fixed that and got Splunk to start at the log on page. I authenticated then got a 500 error.
Not sure why that happened with my Dev versus the Prod instances which went without a hitch. I would like to undo all the certs we installed and take the system back to where it was previously but I did not save the .pem files that were there to begin with. Is there a way to regenerate the ca.pem etc,.. files without doing a reinstall or is there something I missed that could be fixed.
I have not been able to access the Splunk web logs but will have them within the hour.
OK, solved by spending more time searching through Splunk Answers keying on different words.
1st order of business was to tail several different log files to troubleshoot.
splunkd.log revealed that splunk could not find the private key in the server.pem file.
The issue there was a chain that was a link short so to speak.
Because my keys were derived from another, I had to include more in the chain than our other instances of Splunk
This:
cat cert.pem key.pem Int.pem Root.pem > server.pem
created the chain that worked.