Getting Data In

Problem to index the entire CSV file

harry521
New Member

we use csv to track app's performance. I added the csv to forwarder and keep monitoring it. The problem is that while app is running it keeps writing to the CSV, however, only the few minutes at the beginning will be indexed but not the rest. If I move the whole file, while no more update, to another location, I can get everything indexed. Any suggestions?

inputs.conf
[monitor:../logs/performance.csv]
blacklist = -\d
index = test
sourcetype = csv
crcSalt = &ltSOURCE&gt

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You want to index the entire file or just index what's new?? Does your application re-writes the entire file frequently (daily/hourly etc).

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi,
Try to Delete the crcSalt row and restart Splunk.
Remember that to correctly Index csv files, you have to deploy on forwarder also props.conf.
Bye.
Giuseppe

0 Karma

harry521
New Member

Thanks Cusello. I should have noticed the special char being removed when I posted it. It should be "crcSalt = &ltSOURCE&gt". I did try with/without that line. Doesn't help.

I looked into that little bit more, and found out that the data seemed to have been indexed, but, only the first few minutes are correct, and at certain point, the time stamp all messed up which being set months back.

I then created a custom csv sourcetype with specifying the timestamp format and field. That seems to have addressed my problem. I was not thinking of doing that because the timestamp being logged doesn't have millisecond, nor a date.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...