We recently had an issue with one of our indexers. We had to do a restore of /opt/splunk and /var/opt/splunk after rebuilding the OS. When I started the splunkd service, it asked me to accept the license which I thought was strange considering this was a restore of a system that's been in production since 2015. I accepted the license and it proceeded with "upgrading" the config files. After that, the system wasn't recognized by the master node and nor could I get the indexer to rejoin the cluster. I noticed that splunkd failed to run. I re-entered the passkey in clear text for pass4SymmKey in /opt/splunk/etc/system/local/server.conf and attempted to start splunkd again. This time splunkd was able to run, but the indexer couldn't communicate on port 8000 even though in the checking prerequisites it listed port 8000 as open. I got the message "Waiting for web server at https://127.0.0.1:8000 to be available." Also, I got the following error as splunkd was attempting to start when checking conf files for problems -- Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
Couldn't initialize SSL Context for HTTPClient in ServerConfig. Any recommendations on what I should do next to get the indexer to rejoin the cluster?
