All Apps and Add-ons

KVStore mongo db failing to start, after kvstore is restored from backup

splunkIT
Splunk Employee
Splunk Employee

I have restored my kvstore from backup using the steps from here: https://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/BackupKVstore

But now the SHC node's kvstore cannot start. Here are the messages seen in mongod.log:
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] journal dir=/opt/splunk/var/lib/splunk/kvstore/mongo/journal
2018-01-19T20:34:36.865Z I JOURNAL [initandlisten] recover begin
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] info no lsn file in journal/ directory
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover lsn: 0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0
2018-01-19T20:34:36.866Z I JOURNAL [initandlisten] recover applying initial journal section with sequence number 65
2018-01-19T20:34:36.866Z F JOURNAL [initandlisten] dbexception during recovery: 15923 couldn't get file length when opening mapping /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunkvar/lib/splunk/kvstore/mongo/local.0"
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] exception in initAndListen: 15923 couldn't get file length when opening mapping /opt/splunk/var/lib/splunk/kvstore/mongo/local.0 boost::filesystem::file_size: No such file or directory: "/opt/splunk/var/lib/splunk/kvstore/mongo/local.0", terminating
2018-01-19T20:34:36.866Z W REPL [initandlisten] ReplicationCoordinatorImpl::shutdown() called before startReplication() finished. Shutting down without cleaning up the replication system
2018-01-19T20:34:36.866Z I CONTROL [initandlisten] now exiting
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close listening sockets...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to flush diaglog...
2018-01-19T20:34:36.866Z I NETWORK [initandlisten] shutdown: going to close sockets...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: waiting for fs preallocator...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: final commit...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] shutdown: closing all files...
2018-01-19T20:34:36.866Z I STORAGE [initandlisten] closeAllFiles() finished

Tags (1)
0 Karma

scheng_splunk
Splunk Employee
Splunk Employee

looks like it's missing /splunk/var/lib/splunk/kvstore/mongo/local.* file which stores the cluster information.
You may want to check if you have /opt/splunkvar/lib/splunk/kvstore/mongo/local.0 file existing from your kvstore backup.

Also if by any chance you did run "splunk clean kvstore --cluster" command after restoring the kvstore backup, it would wipe out all the local.* files which contains the cluster information.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...