All Apps and Add-ons

JKats Toolkit: How to turn the single event returned with curl into multiple events?

a212830
Champion

Hi,

We are testing out the excellent JKats curl utility, using it to grab some CSV files, and then potentially lookups. The raw feed comes back as one event, and we'd like to turn it into many events. Is there a way to do that? The file being returned is a CSV file. Looks like this:

alt text

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

You'll probably have to use rex to strip the header, then rex to extract the fields... kinda like the image below:

alt text

View solution in original post

0 Karma

Runals
Motivator

try

| curl ... | multikv

This assumes there are field names in a header row.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You'll probably have to use rex to strip the header, then rex to extract the fields... kinda like the image below:

alt text

0 Karma

a212830
Champion

Here's a better image. I wouldn't normally post it as an answer, but the comments won't let me add another image, for some reason...

alt text

The output is a csv with a header and 4 columns. I tried running this search, but it still creates just one entry. I want each entry to be a separate event.

| curl get false "cdcsscrprtpd001.abc.com:8080/ngcc/Reporting/NGCC_VM_Port_Group_Assignments.csv" |eval data="fieldname1, fieldname2, fieldname3" |rex field=data mode=sed "s/fieldname1, fieldname2, fieldname3, //g" |rex field=data "(?\S+), \s+(?\S+), \s+(?\S+)"

0 Karma

jkat54
SplunkTrust
SplunkTrust

I think I can solve this for you now. I can see the image a bit better. So now I just need a few moments to focus. Hang in there...

0 Karma

a212830
Champion

MIght have it - found this link: link text

Testing it now. The search is now:

| curl stream=true get false "myserver.com:8080/e2e/metrics/2016-08-11?server=abcP449&server=labc448&metric=cpu&metric=memory&format=CSV&human=true" | rex max_match=100 field=curl_output "(?<lineData>[^\n]+)" | mvexpand lineData | eval _raw=lineData |fields lineData |table lineData
0 Karma

jkat54
SplunkTrust
SplunkTrust

Yeah its going to have to have something with the max_match in rex for sure.

You shouldnt need that "|eval _raw=lineData" though.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Give this a shot:

 | curl get false "myserver.com:8080/e2e/metrics/2016-08-11?server=abcP449&server=labc448&metric=cpu&metric=memory&format=CSV&human=true"
| rex field=curl_data mode=sed "s/VMHost,VM,PortGroup1,PortGroup2//g" 
| makemv curl_data 
| mvexpand curl_data 
| rex field=curl_data "(?<VMHost>\S+)\,(?<VM>\S+)\,(?<PortGroup1>\S+)\,(?<PortGroup2>\S+)"
| fields VMHost VM PortGroup1 PortGroup2

I actually didnt need a max match, the link you posted gave me another idea with makemv and mvexpand.

You dont need stream=true in your curl command. Thats only if you're passing a data field from splunk search pipeline into your curl command. I updated the "Details" tab of the toolkit app with better instructions. Check it out.

You may have to adjust the rex commands a bit to make it match the data better. I'm not sure if there are spaces between commas or not...

jkat54
SplunkTrust
SplunkTrust

hey @a212830 ... im anxious to know if my search works for you. 😉 You around today?

0 Karma

jkat54
SplunkTrust
SplunkTrust

@a212830, can you come back to this thread and update us please?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Mind accepting the answer or updating it? @a212830

0 Karma

jkat54
SplunkTrust
SplunkTrust

@ppablo_splunk can you close this thread please?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

@jkat54 - whaddya mean by close it? The person who asked the question can accept an answer but that's on @a212830.

0 Karma

jkat54
SplunkTrust
SplunkTrust

It's a year old. He's been asked to come back multiple times. My solution works. Don't hold your breathe waiting on the op.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Besides the curl command is in another app now and has been through some rework...

0 Karma

sloshburch
Splunk Employee
Splunk Employee

You guys beat me to it, but makemv and mvexpand should be used for exactly the purpose of splitting an event into multiple events.

jkat54
SplunkTrust
SplunkTrust

Hey, these look like they are CSV formatted but without a header

The image is blurry but i think it's like this:

fieldvalue, fieldvalue, fieldvalue, fieldvalue

I need a better image of the data... or better sample of the data to help you out here.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Thanks for everyone's support on the curl command. I think I'm going to rework it to use all options instead of keywords. That means you're going to have to specify method=post ssl=true uri=... Etc etc in the future. I recommend that you "subscribe" to the toolkit to get updates as I make them IF you're going to be using this command much. Cheers, ttyl!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...