Getting Data In

How to monitor a file input with the same filename?

szabados
Communicator

I have an input, which is a CSV file. I want to use this as a batch input. The file is generated every day, with the same name.

Most of the time, the beginning of the file is also the same. That's why I think the only method I can achieve to have this ingested every day is to use the CHECK_METHOD = modtime in props.conf
After setting it, I've checked my splunkd.log, I can see that because of the new modtime, Splunk started to read the file, but now I'm getting this messages:

TailReader - Will retry path="" after deferring for 10000ms, initCRC changed after being queued (before=0xd71b3bcce79fb3d8, after=0xb5a12ee23beafb6a).  File growth rate must be higher than indexing or forwarding rate.

I've tried restarting my forwarder, deleting the fishbucket, but I'm still getting this error. (The file wasn't modified during this time)

0 Karma

dgrubb_splunk
Splunk Employee
Splunk Employee

If your csv file has header, you will need to increase the size of the CRC check splunk is doing to identify the file. This can be done using the parameter initCRCLength as indicated in the inputs.conf.spec file:

initCrcLength =
* This setting adjusts how much of a file Splunk reads before trying to
identify whether it is a file that has already been seen. You may want to
adjust this if you have many files with common headers (comment headers, long
CSV headers, etc) and recurring filenames.
* CAUTION: Improper use of this setting will cause data to be reindexed. You
may wish to consult with Splunk Support before adjusting this value - the
default is fine for most installations.
* Defaults to 256 (bytes).
* Must be in the range 256-1048576.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...