How to resume indexing a file after Splunk ignores it?

[joelshprentz]

After splunkd logs an error stating that it is "Ignoring path due to ...", how can we tell Splunk to resume indexing the file at that path?

For the particular mission.log file that an anxious user desires, the ... is "File will not be read, is too small to match seekptr checksum."

Since splunkd logged that message, the mission.log file has grown to 30 MB, rotated to mission.log.1, and been replaced by a new mission.log that has grown to 11 MB. Still, Splunk has not indexed the original mission.log file, the rotated file, or the replacement file (while indexing hundreds of others in different directories). Earlier mission.log files at this path were indexed without problems.

The output of https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus shows this mission file with only a parent attribute. Other files have more attributes such as file position, file size, type, etc.

The (simplified) path the the file is /logs/network/host/mission.log, where there are many networks and hosts. The CRC salt varies for each network to help Splunk distinguish some network metrics files (unrelated to this problem), which may have identical contents.

We are running Splunk 4.1.4 on a Red Hat Enterprise Linux server. Splunk reads most log files via NFS from a NetApp filer.

[jbsplunk · Accepted solution]

This was a problem that was fixed with 4.3.3, Splunk previously ignored files that we'd recorded as having a problem and wouldn't re read the files until after a restart. As of 4.3.3, this is no longer an issue, so you should probably upgrade. If this doesn't solve your issue, you should open a support incident, but it sounds like a good match for the problem you're describing in your post.

[joelshprentz]

Thanks. The tech notes describe the problem we see. The new 4.3.3 functionality should fix it. This will provide some impetus to our request to upgrade from 4.3.1. I accept your answer.

[tpsplunk]

@joelshprentz, i think the SPL# is SPL-44773. http://wiki.splunk.com/Community:SplunkTechNotes

[tpsplunk]

i upgraded my forwarder to 4.3.3 about a month ago and it just experienced the same issue again. 😞 i upgraded to 4.3.4 and opened a case with support.

[joelshprentz]

I must have missed this issue in the 4.3.3 release notes. What is the SPL number?

[tpsplunk]

we have the same problem where periodically a file will be too small after rotation for the seekptr checksum, but then splunk never tries the log again even after its grown to hundreds of MB.