Getting Data In

Why is my tsv data out of order

tkwaller_2
Communicator

Hello

We are parsing data from a TSV source

The data file has a header that is very long, about 281 columns.
What is happening is that we are getting data in the wrong fields.
For example:

Field:
data_poc_technical_name

Values:  
todd.waller@toddwaller.com

this should be the value in the field: data_poc_technical_email.
I also notice that when exporting the data its out of order as well, maybe the issue lies in parsing and configs?
Originally when I tested I got field names in the fields values, I think when data was NULL so I removed FIELD_NAMES from the props and it seemed to have parsed correctly but now doesnt look that way.

These are the props on the UF

[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md_createdAt

and the props from the indexer

[fp:tsv]
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
TIMESTAMP_FIELDS = md_createdAt
KV_MODE = none

Any thoughts?
Thanks for the help!
Todd

0 Karma
1 Solution

tkwaller_2
Communicator

I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.

My edited props on the UF were

[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt

In case it helps anyone else.

View solution in original post

0 Karma

tkwaller_2
Communicator

I figured this out
It seems the issue was timestamping. Once I fixed the timestamp field recognition and reindexed the data it seems to be correct now.

My edited props on the UF were

[fp:tsv]
TIME_FORMAT = %m/%d/%Y %H:%M:%S
FIELD_DELIMITER = \t
HEADER_FIELD_DELIMITER = \t
INDEXED_EXTRACTIONS = TSV
TIMESTAMP_FIELDS = md.createdAt

In case it helps anyone else.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...