Knowledge Management

Is it possible to connect directly to MongoDB?

rharrisssi
Path Finder

I want to maintain a lot of data in my KV Store, but in order to do so I have to keep it clean; but aging out old data.

The problem with:

| inputlookup mylookup | where _time>relative_time(now(),"-7d@h") | outputlookup append=false mylookup 

is that it would cause the full database to be replicated again to other search heads and indexers.

Thus I created a script that will issue delete commands when it runs for any records that are too old. However, it can only interact with the API and delete one entry at a time.

If I were able to connect directly to the MongoDB, I could possible issue a "delete from mytable where _time>value" and it would be 1000% more efficient than deleting one record at a time.

Further, I don't think I can delete records fast enough using Python and the API to keep up with what is being added.

Can anyone shed some light on how I can go about connecting directly to the MongoDB?

1 Solution

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

View solution in original post

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

Lucas_K
Motivator

Awesome!

Took me a little while to figure out the conversion from normal lookup search query to mongodb query.

I got it working with the following.

Normal spl based kv lookup query

|inputlookup summary where LastUpdateTime<1468532752

Mongodb query format ( reference : https://docs.mongodb.com/manual/reference/operator/query/lt/ )

{"LastUpdateTime": {"$lt": 1468532752}}

Curl command url encoded ( http://meyerweb.com/eric/tools/dencoder/ )

curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/summary?query=%7B%22LastUpda...

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You cannot use a Mongodb client to connect to Splunk's KVStore. While it is mongodb, its a modified version to fit within the Splunk framework. This isnt supported.

Lucas_K
Motivator

Did you find a resolution to this?

I'm trying to see it I can use dbconnect with mongojdbc and then schedule a search to run the delete.

http://www.unityjdbc.com/mongojdbc/setup/mongodb_jdbc_splunk.pdf

0 Karma

rharrisssi
Path Finder

I did finally find resolution. The same way you query (GET) the data, you can DELETE.

curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'

You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.

0 Karma

ddrillic
Ultra Champion

The mongodb topics page at mongodb

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...