Getting Data In

Extract some lines of an event from a CSV file and index in separate sourcetype OR Index

mala_splunk_91
Explorer

Hi guys,

Please provide your input on the below scenario.
I have some events like below. Here , I want to extract some part of event which is in CSV format and that is starting after "#" till the end of an event and store them in separate new index/sourcetype, either by using props/transforms conf OR using query.

I have questions like
1)Is there any way to split/extract some part of an event and store it in separate index/sourcetype?
2)How can I extract only CSV event part and display/View it in table format in Splunk?

Final result I need is:
Extract CSV format events separately from the below events and display it in table format OR store in lookup file.(Simply, to make it human readable).

sample.log:

sep-12 02:45:56 This message is received from printer,something like this as a eveent.
sep-12 02:46:56 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xyz,5768,city1,fail,0,0,
mno,7898,city3,done,0,0,
.
.
.
.
.
tno,7459,cityx,done,0,0,
sep-1:3 01:45:56 This message is received from printer,something like this as a event.
sep-1:3 02:05:52 This message is received from printer which is in CSV format....

pname,pcode,plocation,status,header_values,XXX,XXX,XX

Abc,1233,city,done,xxx,0,
xez,5718,city1,fail,0,0,
kno,7878,city3,done,0,0,
.
.
.
.
.
mno,1459,cityx,done,0,0,

Kindly, provide your views.
Thanks
Mala S

1 Solution

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

Configs on Heavy fwd or indexer whichever comes first
props.conf

[YourCurrentSourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER = ([\r\n]+)(?=(\w{3}-\d+\s|\#\w+,))
TRANSFORMS-changesourcetype=csvdata_sourcetype,regular_sourcetype

transforms.conf

[regular_sourcetype]
REGEX = ^\w{3}-\d+\s
FORMAT = sourcetype::RegularLogSourcetype
DEST_KEY = MetaData:Sourcetype

[csvdata_sourcetype]
REGEX = ^\#\w+,
FORMAT = sourcetype::CSVSourcetype
DEST_KEY = MetaData:Sourcetype
0 Karma

mala_splunk_91
Explorer

Thanks Someson, It is working.
And pls tell me how can i show csv event in table format.

0 Karma

somesoni2
Revered Legend

Do you want to store lines starting with "sep-1..." with separate sourcetype (no csv lines) and CSV lines in different sourcetype (split)?

0 Karma

mala_splunk_91
Explorer

Yes, I want to split lines starting with "sept-12.." and csv line and store CSV lines in different sourcetype as CSV, so that i can view data in table format in UI.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...