Getting Data In

How to troubleshoot why 1 indexer in a Splunk indexer cluster crashed and won't restart with a "Bad Decrypt" error?

basher590
Engager

HI,

I have inherited a clustered Splunk setup and I noticed that 1 of my 2 indexers had crashed a couple of days ago.
Trying to restart it yields a Splunk timed out waiting to start error. Looking at the splunkd log I see the following error:

02-22-2016 14:05:35.800 +0000 ERROR SSLCommon - Can't read key file C:\Program Files\Splunk\etc\auth\server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

The key file is there and looks OK to me, though I am not sure how I can test it. I did use the OpenSSL command, but received the same message. I tried changing the password in the config file and I receive a "bad password" error, so I know the PW is correct and it is reading the correct file.

There have been no updates or config changes that I am aware of, this 1 indexer server just seemed to crash.

Is it just a case of creating a new certificate on this one indexer, or are there other steps that need to be followed so I don't break the cluster / indexes?

I am running
Splunk Version
6.2.3
Splunk Build
264376

On Windows 2012 R2 servers.

Thanks

0 Karma
1 Solution

basher590
Engager

I got this fixed in the end by creating a new certificate and applying it to the faulty server.
The first restart worked but I received a new error relating to http://127.0.0.1 instead of https, but after another restart it cleared and all was good.

View solution in original post

0 Karma

basher590
Engager

I got this fixed in the end by creating a new certificate and applying it to the faulty server.
The first restart worked but I received a new error relating to http://127.0.0.1 instead of https, but after another restart it cleared and all was good.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...