After changing DB locations, the KVstore is not initializing in our environment. What can I do to have MongoDB run with Splunk?

[MATTHEW_ORNAWKA]

Having an issue with the KVstore not initializing in our environment. The error log from mongod.log is below

I have attempted a ./mongod --repair, but this hasn't made a difference. This all started after changing DB locations and cleaning all databases.

 2016-10-24T16:35:15.302Z I STORAGE  [initandlisten] _getOpenFile() invalid file index requested 1
 2016-10-24T16:35:15.302Z I -        [initandlisten] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201
 2016-10-24T16:35:15.319Z I CONTROL  [initandlisten]
  0x7fbe5f351062 0x7fbe5f2eb7e9 0x7fbe5f2ce889 0x7fbe5f12154f 0x7fbe5f1215b2 0x7fbe5f120ce8 0x7fbe5f12e2b3 0x7fbe5f11a153 0x7fbe5f11aa60 0x7fbe5f11b22e 0x7fbe5f11e608 0x7fbe5ecc8876 0x7fbe$
 ----- BEGIN BACKTRACE -----
 {"backtrace":[{"b":"7FBE5E689000","o":"CC8062"},{"b":"7FBE5E689000","o":"C627E9"},{"b":"7FBE5E689000","o":"C45889"},{"b":"7FBE5E689000","o":"A9854F"},{"b":"7FBE5E689000","o":"A985B2"},{"b$
  mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x7fbe5f351062]
  mongod(_ZN5mongo10logContextEPKc+0xE9) [0x7fbe5f2eb7e9]
  mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB9) [0x7fbe5f2ce889]
  mongod(_ZNK5mongo19MmapV1ExtentManager12_getOpenFileEi+0x8F) [0x7fbe5f12154f]
  mongod(_ZNK5mongo19MmapV1ExtentManager9getExtentERKNS_7DiskLocEb+0x22) [0x7fbe5f1215b2]
  mongod(_ZN5mongo19MmapV1ExtentManager9cacheHintERKNS_7DiskLocERKNS_13ExtentManager8HintTypeE+0x18) [0x7fbe5f120ce8]
  mongod(_ZN5mongo19CappedRecordStoreV1C2EPNS_16OperationContextEPNS_28CappedDocumentDeleteCallbackERKNS_10StringDataEPNS_21RecordStoreV1MetaDataEPNS_13ExtentManagerEb+0xE3) [0x7fbe5f12e2b$
  mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry14_insertInCacheEPNS_16OperationContextERKNS_10StringDataENS_8RecordIdEPNS0_5EntryE+0x213) [0x7fbe5f11a153]
  mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntry5_initEPNS_16OperationContextE+0x6D0) [0x7fbe5f11aa60]
  mongod(_ZN5mongo26MMAPV1DatabaseCatalogEntryC1EPNS_16OperationContextERKNS_10StringDataES5_bb+0x18E) [0x7fbe5f11b22e]
  mongod(_ZN5mongo12MMAPV1Engine23getDatabaseCatalogEntryEPNS_16OperationContextERKNS_10StringDataE+0x1B8) [0x7fbe5f11e608]
  mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x126) [0x7fbe5ecc8876]
  mongod(_ZN5mongo13initAndListenEi+0x107D) [0x7fbe5eba3c3d]
  mongod(main+0x159) [0x7fbe5eba8539]
  libc.so.6(__libc_start_main+0xFD) [0x7fbe5d56ccdd]

any advice will be welcome

[mornawka]

No solution that saved the data no... essentially I had to wipe it all and start from scratch. Mine was a lab instance, and the data was non critical.

sorry I didn't have a better answer.

[jacobsmith14]

I'm seeing the same issue. Ever find a resolution?

[dvg06]

I was able to fix similar issue by changing directory/file permissions.

770 for kvstore, mongo, dump directories. Mongo and dump are sub directories of kvstore.
600 for splunk.key file inside mongo directory.

[aaraneta_splunk]

Hi @MATTHEW.ORNAWKA@TELUS.COM - Are you using the MongoDB Monitoring App? Just want to make sure your post is tagged properly.

[MATTHEW_ORNAWKA]

thank you for your response. I am having an issue with the KVStore in splunk not starting, and this is where I have ended up. I am not using the MongoDB Monitoring App. MongoDB is not currently running.