Can you help me with the following KV Store error: "Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty"

[paola92]

Hi,

I have several errors related to KV Store as:

-Failed to start KV Store process. See mongod.log and splunkd.log for details.
-KV Store changed status to failed. KVStore process terminated.
-KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

So when reviewing the mongod.log I see:

Detected unclean shutdown - /home/dbindex/kvstore/mongo/mongod.lock is not empty.
 I STORAGE  [initandlisten]
 I STORAGE  [initandlisten] ** WARNING: Readahead for /home/dbindex/kvstore/mongo is set to 4096KB
 I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 I STORAGE  [initandlisten] **************
 old lock file: /home/dbindex/kvstore/mongo/mongod.lock.  probably means unclean shutdown,
 but there are no journal files to recover.
 this is likely human error or filesystem corruption.
 please make sure that your journal directory is mounted.
 found 76 dbs.
 see: http://dochub.mongodb.org/core/repair for more information

And, I tried to run the ./mongod --dbpath /DB/kvstore/mongo --repair and obtained the error:

./mongod: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

So I need help to solve the problem!

[rajan_kumar_rai]

We have to delete the mongod.lock file from the /Application/Splunk/var/lib/kvstore/mongo —> mongod.lock

We have to remove this file to done that simply run below command in that path

# SUDO rm -rf mongod.lock >> it will delete the that file.

Next we have to provide the read permission to SPLUNK.KEY file to done that run below command

# CHMOD 400 SPLUNK.KEY ( IT WORKS ) 

Then restart the splunk from GUI. Go to settings > server controls > Restart 😀

[reswob4 · Accepted solution]

The answer by @claudio.manig in this thread https://answers.splunk.com/answers/236495/splunk-kv-store-does-not-start.html helped me.

After verifying all the permissions discussed were set correctly on my server (linux), I stopped Splunk, deleted the mongod.lock file and restarted splunk and the error went away. I didn't need to run the repair utility.