Getting Data In

Is the file CRC on a Forwarder unique to the input? Can I change input method through partial ingestion?

mcrawford44
Communicator

We have some customers indexing recovery data from a data outage. These files are 15-30 minutes of logging each. Up to several GB.

Thus far they have been using a standard monitor. But have been pulling files out of the monitor folder. They were "guessing" when Splunk was finished indexing instead of validating with event counts. I have checked, and some of the files were partially ingested.

I want to move them to a batch monitor, but I have questions;

  • Will these files be re-indexed fully, or will they resume based on CRC?
  • If a file has already been fully indexed with the standard monitor, will it be skipped if moved to the batch folder?
  • Is the CRC unique to each input, or can it be used for all inputs at any time?
  • If they will not resume, how would you suggest we remediate the issue without duplicate events?

Thanks in advance!

0 Karma
1 Solution

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

View solution in original post

0 Karma

mcrawford44
Communicator

The answer is;

CRC appear to be unique to a monitor. Moving the files in anyway to a new monitor path will result in the re-indexing of that file. No resumes.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...