Getting Data In

When import TSV file, a field end with '\' will be combined with the following field wrongly

ryu_kahou
Explorer

I'm importing tab-delimited files formatted as the following. The space is tab.

 "field1    field2    field3    field4"

The files are imported with command "splunk add oneshot " and parsed with the following definition in transforms.conf.

DELIMS = "\t"
FIELDS = "field_name1","field_name2","field_name3","field_name4"

It works fine but for some line the value goes into wrong field name. It took me a lot of time to find the reason. I found that if a field ends with '\', it will be combined with its next field.

For example, "field1 field2\ field3 field4" will be parsed as

field_name1=field1
field_name2=field2\ field3
field_name3=field4
field_name4=

If I remove all ending '\' it will work fine. But I have to import the original data as is. Is there any way to import field end with '\'?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The backslash is escaping the tab so it is treated as an ordinary character rather than as a separator. One solution is to use a regex. Replace the DELIMS and FIELDS lines in your transforms.conf with this line:

REGEX=(?<field_name1>.*?[\\]?)\t(?<field_name2>.*?[\\]?)\t(?<field_name3>.*?[\\]?)\t(?<field_name4>.*?)

If the backslash is not part of the field value then use this REGEX string:

REGEX=(?<field_name1>.*?)[\\]?\t(?<field_name2>.*?)[\\]?\t(?<field_name3>.*?)[\\]?\t(?<field_name4>.*?)
---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The backslash is escaping the tab so it is treated as an ordinary character rather than as a separator. One solution is to use a regex. Replace the DELIMS and FIELDS lines in your transforms.conf with this line:

REGEX=(?<field_name1>.*?[\\]?)\t(?<field_name2>.*?[\\]?)\t(?<field_name3>.*?[\\]?)\t(?<field_name4>.*?)

If the backslash is not part of the field value then use this REGEX string:

REGEX=(?<field_name1>.*?)[\\]?\t(?<field_name2>.*?)[\\]?\t(?<field_name3>.*?)[\\]?\t(?<field_name4>.*?)
---
If this reply helps you, Karma would be appreciated.

ryu_kahou
Explorer

Thank you for your answer.

I think it's a bug of DELIMS. Even simple split function doesn't have the problem.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...