Knowledge Management

Why does kvstore lookup record insert fail with error "An error occurred during the last operation ('createCollection', domain: '5', code: '10334')"?

sat94541
Communicator

When user try to update kvstore using command like

| inputlookup [backupdata]  | output 

This errors out with error below

Error :
Could not write to collection [kvstore] error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101. 

Preliminary research indicates this could be a database related error.

3-08-2017 08:39:21.322 ERROR KVStorageProvider - An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101
03-08-2017 08:39:21.322 ERROR outputcsv - sid:1488991161.995 Could not write to collection 'vulnerability_management': An error occurred during the last operation ('createCollection', domain: '5', code: '10334'): exception: BSONObj size: 1751347969 (0x68637301) is invalid. Size must be between 0 and 16793600(16MB) First element: d_time: ?type=101.
Tags (2)
0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

View solution in original post

abhullar_splunk
Splunk Employee
Splunk Employee

Running Step 6. I was seeing:

error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory

Basically you are trying to run mongo flat from the file system but the dependencies (like openssl) arent install on the SYSTEM, theyre only installed in SPLUNK.

Either the lib file needs to be installed under default path /usr/lib or the LIB path needs to be set as below
export LD_LIBRARY_PATH=/opt/splunk/lib/:$LD_LIBRARY_PATH

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

Try steps below

1.Stop splunk
2. Backup the folder $SPLUNK_DB/kvstore/mongo and move the backup somewhere outside Splunk.
3. Invoke: $SPLUNK_HOME/bin/splunk clean kvstore --cluster
4. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal (as it's not empty and it may cause some issues).
5. Manually remove file $SPLUNK_DB/kvstore/mongo/mongod.lock .
6. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 —repair
Please collect the console output from repair; it should exit with no errors if successful.
7. Start splunk.

immortalraghava
Path Finder

What happens to the backup ? In what scenarios it will be useful ?

0 Karma

sat94541
Communicator

this resolve the issue

0 Karma

mbrunetto
Path Finder

had a similar issue, this fixed it. Thanks!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...