Deployment Architecture

upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

sylim_splunk
Splunk Employee
Splunk Employee

Splunk start produces error below and stopped running. So far it failed on 2 servers in a row.
Any suggestions?


Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
license with stackid=enterprise found in stack download-trial
license with stackid=enterprise found in stack download-trial
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 14, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.

ERROR while running mongod-fix-voting-priority migration.

There is no splunkd.log messages put in the file since the upgrade and even 'splunk diag' fails with the same error.

Tags (1)
1 Solution

sylim_splunk
Splunk Employee
Splunk Employee

We have been using default certs and were able to work around it by re-create the expired server.pem by following the steps below;

$ splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n SplunkServerDefaultCert
$ mv server.pem server.pem.orig
$ mv SplunkServerDefaultCert.pem server.pem

The Mongod.log has had error messages against expired server.pem file since last year so this was the last thing we suspected - some changes in 7.2 might have started to cause the behavior.

To check the expire date;
$ openssl x509 -noout -enddate -in $SPLUNK_HOME/etc/auth/server.pem

If it is still the same then check the below;
i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

View solution in original post

saramamurthy_sp
Splunk Employee
Splunk Employee

I had a similar case where the client himself found the solution, he performed the below steps and it fixed the issue.

While upgrading from 6.6 to 7.2.3 if you are not able to start the Splunk and if it keeps on prompting to accept license follow the below steps

Go to /opt/splunk
mv ftr /tmp
cd /etc/init.d
./splunk start

If the above steps are followed we can get rid of MongoDB error and we can upgrade to the latest version.

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

We have been using default certs and were able to work around it by re-create the expired server.pem by following the steps below;

$ splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n SplunkServerDefaultCert
$ mv server.pem server.pem.orig
$ mv SplunkServerDefaultCert.pem server.pem

The Mongod.log has had error messages against expired server.pem file since last year so this was the last thing we suspected - some changes in 7.2 might have started to cause the behavior.

To check the expire date;
$ openssl x509 -noout -enddate -in $SPLUNK_HOME/etc/auth/server.pem

If it is still the same then check the below;
i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

marrette
Path Finder

I also had this same error when upgrading from Splunk 6.5 to 7.3.2. I tried Sylim's fixes and they didn't seem to help. But then I found the following solution which fixed it immidately (though I have a feeling I would have needed to have applied Sylim's fix also):

The permissions on $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key were -rw-rw----
I've seen it documented somewhere that this file must be set to owner read only (-r--------)
I ran the following to fix: chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
Then re-ran the Splunk upgrade successfully.

arrangineni
Path Finder

Worked like a charm. Thank you @marrette

0 Karma

333mendyp
Loves-to-Learn Lots

Mine was a permission issue, the above steps from Marrette fixed our issue and I was able to upgrade and restart my IDX.

PS: In my environment, our "splunk.key" was located @ /splunk/kvstore/mongo/splunk.key.

Thank you.

0 Karma

witski
Explorer

I have encountered the same ERROR when I tried to upgrade my cluster from 7.1.4 to 7.1.5.
I have a two-site indexer-cluster. Several indexers in site2 produced "mongod exited abnormally (exit code 14, ..." and "ERROR: Failed to running mongod-fix-voting-priority migration." during starting procedure.
These indexers have one thing in common. The modtime of the server.pem is in May 2016, and now is in Jan 2019.
I tried to create new server.pem with commands provided by your answer and it worked.
After I replaced the old server.pem file with the new one, Splunk started normally.

MoniM
Communicator

Hi Sylim,
i also faced the same issue. i followed the above steps but the issue still persists.
[root@localhost ~]# openssl x509 -noout -enddate -in /opt/splunk/etc/auth/server.pem
notAfter=Jan 9 08:43:08 2022 GMT

the above is my output after creating new certificate. so when i am trying to run this command ./splunk start --accept-license ,the output is below:-
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
ERROR while running mongod-fix-voting-priority migration.

can you please help me out here?

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Could you check the below?

 i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
 ii) expired cert (KV store was not used at all)

 iii) unsupported ciphers 
 iv) mongod port taken by other process,

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...