Getting Data In

How to convert a single event into an outputlookup CSV file?

koshyk
Super Champion

We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg

[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime

Example message

hostname,user
host1,user1
host2,user2
host3,user3

If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"

What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
0 Karma
1 Solution

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv

View solution in original post

0 Karma

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv
0 Karma

woodcock
Esteemed Legend

I am glad to heilp....how exactly did I help?

0 Karma

koshyk
Super Champion

One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...