Importing a CSV file as a threat intelligence lookup list, how can I test that this file is being parsed against my data?

[santorof]

I just completed importing a CSV file as a threat intelligence lookup list. I followed Splunk documentation (6.2) and one of the details for threats by IP address is that IPs and descriptions are required. The file I was given only had IP addresses, so I changed under parsing options to only look at the first column for IPs with nothing defining descriptions. My question is, how can I test that this threat intelligence document is being parsed against my data and would files only with IPs still function without a description field?

[sheamus69]

This is how I would do it (assuming that an existing threat intel field exists, called the-threat-intel-list.csv):

1. Upload "Update.csv" into search app
2. Change permissions on Update to App level
3. Once App viewable, you will be able to delete Update once done
4. Craft Splunk search to load in previous threat list, then append on the Update file (assuming updates go on the end) - this will NOT make any permanent changes at this point, merely display the output : | inputlookup the-threat-list.csv | inputlookup append=t Update.csv 4a. Dedup on certain columns, if desired (quoted only if special characters, such as periods or spaces, in field names)
5. Inspect search results to ensure columns match up properly (generally not a problem, in Splunk, beware if updating a CSV from python)
6. When confident list is as desired, write out to file: | inputlookup the-threat-list.csv | inputlookup append=t Update.csv | dedup [field to dedup if desired] | outputlookup the-threat-list.csv
7. Inspect file was written as intended: | inputlookup the-threat-list.csv

[rashid47010]

HI Ninjas,

I upload custom threat intelligence. file_name, description,url
the threat activity detected notable is triggering successfully against custom threat intelligence that I have upload previously.
Now I want to remove those threat intelligence feeds. that threat activity detected should not trigger against that custom feeds.

[santorof]

I did the above but have more questions. The threat list(the-threat-list) does show all the IP's I want to filter out by. So my logic is comparing these ip's in the IP field to a field called IP in my sourcetype firewall for example(checking for hits manually). Would I use the diff command to compare the IP field from the-threat-list to my sourcetype firewall field=IP for any matches? Just not sure how to go about it. I also tried to do the threatlookup command with files that are being pulled down from a url but got no results.

[woodcock]

Have you seen the GetWatchList app?

https://splunkbase.splunk.com/app/635/

[santorof]

I looked at it but was not looking to add any apps that weren't built by splunk or splunk supported. Would I be able to install this app on my local machine and extract the information I would need to compare my data against specific threat lists?

[woodcock]

This anti-open-source bias is silly and severely limiting, but it is your system.

[woodcock]

Yes, that is the whole point; you give it the URL where it can get the threatlist and it does the rest.

[santorof]

Sorry I did not make it clear. I have a local copy of splunk with no data I use as my test environment. The real data and infrastructure is what I don't want to install the app on. If it was xml or dashboard logic I could extract it from my local version but because its a command I dont believe it will help me with the real data.