I am indexing very large files each day, each on the order of 20+GB. I am using [batch] and move_policy = sinkhole such that the file is read, indexed and intentionally deleted. However, sometimes the # of events indexed are less than the # of events in the file. Here is the inputs.conf segment that applies. [batch:///my_path_to_the_file/*.import] move_policy = sinkhole sourcetype = my_sourcetype index = my_index crcSalt=<SOURCE> disabled = false These large files are being SFTP'd to the Heavy Forwarder / Dropbox and the transfer can take 15+ minutes to complete. I am wondering whether the [batch] process will take a snapshot of the file and index it sometime after it arrives but before the transfer has completed. I am presuming that [batch] only looks at the file once. Essentially, can what I attempt to show in the following image actually occur?
... View more
