Knowledge Management

How to directly connect to kvstore using a mongodb client and bulk fill my kvstore with 3,000,000 entries?

bjoernjensen
Contributor

Hi there,

I would like to initially bulk fill my kvstore with around 3.000.000 entries.

AFAIK the REST API allows to add one entry at a time. Not taking the overhead into account, this takes way too long.

Therefore, I need a direct connection to the kvstore (mongo db 2.6.7-splunk). My first task then would be: "getting a list of all database objects (show dbs)".

I tried using mongo shell 3.0.3. The connection could be established (to the db admin). But I am not allowed to do:

show db
"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0 }",
"code" : 13

But I am connected not using any pem/key file.

With node.js and the package mongodb, I use these options, since I found them in mongod.log:

options: { net: { port: 8191, ssl: { PEMKeyFile: "xxx\etc\auth\server.pem", PEMKeyPassword: "<password>", mode: "preferSSL" } }, replication: { oplogSizeMB: 1000 }, security: { keyFile: "xxx\var\lib\splunk/kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0" }, storage: { dbPath: "xxx\var\lib\splunk/kvstore\mongo", smallFiles: true }, systemLog: { timeStampFormat: "iso8601-utc" } }

But same with this approach:

AssertionError: null == {"name":"MongoError","message":"not authorized on admin to execute command { listDatabases: 1 }","ok":0,"errmsg":"not authorized

My questions so far:

1 How can I bulk fill the kvstore?

2 How can access the kvstore directly?

3 (How) can I use my own mongodb as kvstore?

Thanks - Björn

Tags (2)
1 Solution

dgladkikh_splun
Splunk Employee
Splunk Employee
  1. Two options: a) you can use batch_save REST call http://docs.splunk.com/Documentation/Splunk/6.2.3/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2... It is limited by default to maximum of 1000 documents, so in your case it will take 3,000 calls.

b) you can create a lookup for this collection, save your data into CSV file and migrate it from CSV into your kvstore collection by issuing a search like "| inputlookup myfile.csv | outputlookup mykvstorecollection_lookup"

2 & 3. That is not supported. Even if you will find out how to make 2 and 3 work - there are will be a huge chance that not everything will work as you expect or something can be broken with following updates. Please let us know if you have some other reasons (other than #1) why would you like to have #2 and #3.

View solution in original post

dgladkikh_splun
Splunk Employee
Splunk Employee
  1. Two options: a) you can use batch_save REST call http://docs.splunk.com/Documentation/Splunk/6.2.3/RESTREF/RESTkvstore#storage.2Fcollections.2Fdata.2... It is limited by default to maximum of 1000 documents, so in your case it will take 3,000 calls.

b) you can create a lookup for this collection, save your data into CSV file and migrate it from CSV into your kvstore collection by issuing a search like "| inputlookup myfile.csv | outputlookup mykvstorecollection_lookup"

2 & 3. That is not supported. Even if you will find out how to make 2 and 3 work - there are will be a huge chance that not everything will work as you expect or something can be broken with following updates. Please let us know if you have some other reasons (other than #1) why would you like to have #2 and #3.

bjoernjensen
Contributor

Thanks for you input on this. I actually missed the batch_save endpoint.

For #2 and #3: One could use the kv store as an interface to master data systems. Moreover in a scenario where (critical) process data is stored within the kv store the customer might be interested in having a non-proprietary access to this kind of data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...