Multiple log files with several keys

[renaudleroy]

Hi all!

I've got different log files (in fact, extracts from different databases) from a data warehouse (abstractly a big database) :
ex:
database1 (asset management) give :
- a list of asset_management_computers
- a list of vulnerabilities for these computers
- maybe a different file with vulns ids and details

database2 (antivirus) gives :
- a list of computers for the antivirus product
- a list of infections

The goal is to be able to have information for a source IP, or a hostname, and extract the results (vulnerabilities and mapped potential AV exploiting these vulnerabilities) from splunk.

The problem is that there are different Ids for individual computers (not the same ID for asset management and for AV) and cross-link IDs, I mean: a vulnerability is identified and detailed in a 3rd file, but the ID vulnerability is present in the extracted list of vulnerabilities and the AV has different keys present in different files. So i need to find a LINK to map between these different reference and goal is to search e.g for an IP and find the corresponding vulnerabilities and virus alertes that tried to exploit them.
You see ?

My approach was atm to index (in different indexes) the extracted files from databases :
- create an index named asset_management_computers
- create an index named asset_management_vulns
- create and index named asset_management_vulns_details

• create an index named av_computers
• create an index named av_infections

Then inject logs directly in the corresponding indexes.
And for search e.G i search for all infos referring to an IP source :
index = asset_management_* or index = av_* 192.168.0.1

I'll thus be able to find the results corresponding to events for the indexed files (vulns, ids, av infeections..etc)

Is there any other method for the LINK between files, and for search enhancement ?

Thansk for your answers !

[yannK]

Check the join command to link results from 2 searches over a field, or if you have a database define a database lookup.

join : http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Join
lookup to static file : http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Lookup
database lookup using dbconnect : http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Abouttheconnector

Otherwise another solution is to use a subsearch to return conditions for the main search, but it is better suited for small sub sets of events than all of your ips ....
see http://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutsubsearches

[renaudleroy]

The question is indeed about a key present in one file, referencing a vuln id, on which details are present in another file with this key, about a computer with another key, for an IP, on which a different key is present in another table/file, refering to another key/ID... you see ? 🙂

Is there any other method for the LINK between files, and for search enhancement ?

Thansk for your answers !