I want to maintain a lot of data in my KV Store, but in order to do so I have to keep it clean; but aging out old data.
The problem with:
| inputlookup mylookup | where _time>relative_time(now(),"-7d@h") | outputlookup append=false mylookup
is that it would cause the full database to be replicated again to other search heads and indexers.
Thus I created a script that will issue delete commands when it runs for any records that are too old. However, it can only interact with the API and delete one entry at a time.
If I were able to connect directly to the MongoDB, I could possible issue a "delete from mytable where _time>value" and it would be 1000% more efficient than deleting one record at a time.
Further, I don't think I can delete records fast enough using Python and the API to keep up with what is being added.
Can anyone shed some light on how I can go about connecting directly to the MongoDB?
I did finally find resolution. The same way you query (GET) the data, you can DELETE.
curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'
You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.
I did finally find resolution. The same way you query (GET) the data, you can DELETE.
curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'
You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.
Awesome!
Took me a little while to figure out the conversion from normal lookup search query to mongodb query.
I got it working with the following.
Normal spl based kv lookup query
|inputlookup summary where LastUpdateTime<1468532752
Mongodb query format ( reference : https://docs.mongodb.com/manual/reference/operator/query/lt/ )
{"LastUpdateTime": {"$lt": 1468532752}}
Curl command url encoded ( http://meyerweb.com/eric/tools/dencoder/ )
curl -k -u admin:changeme -X DELETE https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/summary?query=%7B%22LastUpda...
You cannot use a Mongodb client to connect to Splunk's KVStore. While it is mongodb, its a modified version to fit within the Splunk framework. This isnt supported.
Did you find a resolution to this?
I'm trying to see it I can use dbconnect with mongojdbc and then schedule a search to run the delete.
http://www.unityjdbc.com/mongojdbc/setup/mongodb_jdbc_splunk.pdf
I did finally find resolution. The same way you query (GET) the data, you can DELETE.
curl -k -u myuser:mypass -X DELETE 'https://localhost:8089/servicesNS/nobody/myapp/storage/collections/data/mykvstoret?query={"_time":{"...'
You may have to escape/convert some of the chars in the above cURL command for it to work- { is %7B, } is %7D and $ is %24. epoch_time is obviously meant to be an integer.
The mongodb topics page at mongodb