Splunk Search

Transform to remove the first column of a CSV if it matches a set of criteria?

dennywebb
Path Finder

i have logs coming in as CSV files, but sometimes junk data is truncated on the front by the system generating them, sometimes not. they are otherwise identical. i have no control on this system.

the upside is the junk is always of a certain pattern:
blah:blah,gooddata,gooddata,gooddata,gooddata,gooddata
versus the clean ones which are just:
gooddata,gooddata,gooddata,gooddata,gooddata

how do i get a transform to drop that first column before indexing ONLY IF it has X:X as a value?

been beating my head against this for 2 days... regex is not my strong point.

Tags (3)
1 Solution

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

View solution in original post

Ayn
Legend

Well a regex that will match anything at the beginning of the line until the first comma, with a : character in it would look like this:

^[^,]+:[^,]+,

dennywebb
Path Finder

an update, installed and working perfectly.

0 Karma

dennywebb
Path Finder

you sir/mam, are the man/woman. the end result:
^([^,]+:[^,]+,)?(.*)
works great. i should be able to just feed $2 back into the raw for either type and always have the same result.

0 Karma

Ayn
Legend

Sorry, forgot two + signs in my regex. Editing my answer with a correct regex.

0 Karma

dennywebb
Path Finder

though when i put what you provided into http://regexlib.com/RETester.aspx
as: ^[^,]:[^,],(.*)
with data as: something:anything,stuff1,stuff2,stuff3:stuff4,stuff5

nothing comes back.

0 Karma

dennywebb
Path Finder

and then i would use the Dest_Key=_Raw in the transform stanza to replace the raw log with my newly cleaned one i presume?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...