Is there a clever way to index mongodb data without a rising columns into Splunk

[bestSplunker]

hello. I use splunk db connect 3.1.3 connect mongodb database. it is working now. and I can use SQL statement query data from mongodb

|dxquery connection="testmongodb" query="select * from result" .

We all know that mongodb does not contain a self-incrementing column. So mongodb's data is similar to the following format:

info    ip  port    task_date   task_id time    vul_info
Unauthorized Access 172.16.10.9 6379    2019-6-6    d40617172258939a57fdb5617724fc55    2019-6-6    {"vul_type":"Weak password",vul_name:"Redis Weak password",vul_level:"High"}
SMB Remote Overflow 10.10.2.8   445 2019-6-6    cfab842aa0e8166cabb2f4548477756b    2019-6-6    {"vul_type":"Remote Overflow",vul_name:"SMB Remote Overflow",vul_level:"High"}
MySQL Weak password 10.10.2.7   3306    2019-6-13   2389ccda6788fc124d1cec7a951f7089    2019-6-13   {"vul_type":"Weak password",vul_name:"MySQL Weak password",vul_level:"High"}

Firstly, it does not have an rising column, for example id, secondly, it does not have a timestamp.

If I use input with batch to index these data into splunk, there will be a lot of duplicate data. So I hope. Every time there is new data in the result collections, it can automatically index to Splunk .

If this mongo collection have a rising column, it will be easy to implement this requirement, unfortunately not.

So is there a clever way to index new data from mongodb to splunk?

[DavidHourani · Accepted solution]

Hi @bestSplunker,

Have you considered using the kv-store...it's also a mongodb data base so you can just dbquery your data from your DB into the kv-store and then use the results from there.

Cheers,
David

[bestSplunker]

@DavidHourani

what should I do? Is there a link to the documentation? These data are vulnerability information, meaning that a host has discovered a new vulnerability, so I want alert it if result collection adding a new recorder

[DavidHourani]

sure, have a look here for how to set up the kv-store : https://docs.splunk.com/Documentation/Splunk/7.3.0/Knowledge/ConfigureKVstorelookups
It's very easy to use, and once the data is there you can use it as a lookup, which is perfect for vulnerability data. You'll be able to use that data to enrich your logs using the lookup command.
Let me know when you try setting it up if you find any difficulty, i'd be happy to help.

[bestSplunker]

Thank you, I solved this problem through kvstore. firstlly, I query the result from mongodb, and then search for it in Kvstore. If the result does not exist in kvstore, an alert is triggered, finally fill all the results into the kvstore..

|dbxquery connection="testmongodb" query="select * from Result"
    |search NOT [|inputlookup resultcollections]
    |outputlookup resultcollections