Hi,
We are testing out the excellent JKats curl utility, using it to grab some CSV files, and then potentially lookups. The raw feed comes back as one event, and we'd like to turn it into many events. Is there a way to do that? The file being returned is a CSV file. Looks like this:
You'll probably have to use rex to strip the header, then rex to extract the fields... kinda like the image below:
try
| curl ... | multikv
This assumes there are field names in a header row.
Here's a better image. I wouldn't normally post it as an answer, but the comments won't let me add another image, for some reason...
The output is a csv with a header and 4 columns. I tried running this search, but it still creates just one entry. I want each entry to be a separate event.
| curl get false "cdcsscrprtpd001.abc.com:8080/ngcc/Reporting/NGCC_VM_Port_Group_Assignments.csv" |eval data="fieldname1, fieldname2, fieldname3" |rex field=data mode=sed "s/fieldname1, fieldname2, fieldname3, //g" |rex field=data "(?\S+), \s+(?\S+), \s+(?\S+)"
I think I can solve this for you now. I can see the image a bit better. So now I just need a few moments to focus. Hang in there...
MIght have it - found this link: link text
Testing it now. The search is now:
| curl stream=true get false "myserver.com:8080/e2e/metrics/2016-08-11?server=abcP449&server=labc448&metric=cpu&metric=memory&format=CSV&human=true" | rex max_match=100 field=curl_output "(?<lineData>[^\n]+)" | mvexpand lineData | eval _raw=lineData |fields lineData |table lineData
Yeah its going to have to have something with the max_match in rex for sure.
You shouldnt need that "|eval _raw=lineData" though.
Give this a shot:
| curl get false "myserver.com:8080/e2e/metrics/2016-08-11?server=abcP449&server=labc448&metric=cpu&metric=memory&format=CSV&human=true"
| rex field=curl_data mode=sed "s/VMHost,VM,PortGroup1,PortGroup2//g"
| makemv curl_data
| mvexpand curl_data
| rex field=curl_data "(?<VMHost>\S+)\,(?<VM>\S+)\,(?<PortGroup1>\S+)\,(?<PortGroup2>\S+)"
| fields VMHost VM PortGroup1 PortGroup2
I actually didnt need a max match, the link you posted gave me another idea with makemv and mvexpand.
You dont need stream=true in your curl command. Thats only if you're passing a data field from splunk search pipeline into your curl command. I updated the "Details" tab of the toolkit app with better instructions. Check it out.
You may have to adjust the rex commands a bit to make it match the data better. I'm not sure if there are spaces between commas or not...
hey @a212830 ... im anxious to know if my search works for you. 😉 You around today?
@a212830, can you come back to this thread and update us please?
Mind accepting the answer or updating it? @a212830
@ppablo_splunk can you close this thread please?
@jkat54 - whaddya mean by close it? The person who asked the question can accept an answer but that's on @a212830.
It's a year old. He's been asked to come back multiple times. My solution works. Don't hold your breathe waiting on the op.
Besides the curl command is in another app now and has been through some rework...
You guys beat me to it, but makemv
and mvexpand
should be used for exactly the purpose of splitting an event into multiple events.
Hey, these look like they are CSV formatted but without a header
The image is blurry but i think it's like this:
fieldvalue, fieldvalue, fieldvalue, fieldvalue
I need a better image of the data... or better sample of the data to help you out here.
Thanks for everyone's support on the curl command. I think I'm going to rework it to use all options instead of keywords. That means you're going to have to specify method=post ssl=true uri=... Etc etc in the future. I recommend that you "subscribe" to the toolkit to get updates as I make them IF you're going to be using this command much. Cheers, ttyl!