Hello,
I would like to know if it was possible to send a CSV to the HEC, and to take into consideration the names of the columns in the index. Here is my CSV file:
url,vuln,fix,severity,package,imageTag
https://security-tracker.debian.org/tracker/CVE-2018-1302,CVE-2018-1302,None,Unknown,apache2-2.4.25-...
https://security-tracker.debian.org/tracker/CVE-2007-3303,CVE-2007-3303,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1580,CVE-2003-1580,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2003-1581,CVE-2003-1581,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2008-0455,CVE-2008-0455,None,Negligible,apache2-2.4....
https://security-tracker.debian.org/tracker/CVE-2001-1534,CVE-2001-1534,None,Negligible,apache2-2.4....
I would like the url,vuln,fix,severity,... headers to be recognized as a column in splunk, to get a table.
When I manually import my CSV file the fields are well recognized and everything works as I want. But with HEC, the headers line is simply added to the event list.
To send my CSV to HEC, I use a python script and the PyHEC modue (github jonromero pyHEC) with this piece of code I send line by line the content of my CSV:
with open("vuln.csv") as fp:
for line in fp:
print hec.send(line)
I also tried without a loop, but the whole CSV content is stored in a single event.
I also tried field extraction, but I send 2 different CSVs in the same index, with different fields.
I specify that this works with CSV files sent manually to splunk.
Do you have an idea to make splunk recognize CSV headers when sending via HTTP Event Collector?
Sincerely.
What a great idea. I've been testing having users upload to a Heavy Forwarder, and then monitoring the app folder they upload to ingest the csv as a log instead of a lookup, however your approach seems much more pragmatic.
@skhedim Hey, I am facing the same problem. Did you find a solution for it?
Use a dict reader in python to read it in as a list of dicts.
Then send as the event dict.
https://github.com/georgestarcher/Splunk-Class-httpevent