Deployment Architecture

Splunk forwarder Skipping Log files occasionally

vaibhavagg2006
Communicator

HI Experts
I am having an issue in indexing the log file which gets rotated ever hour. The log file error.log gets rotated every hour at top of the hour and a new file is created with the same name(error.log). The old file gets renamed and zipped to error.log._timestamp.gz.
Sometimes splunk does not index the file for an hour and resumes the indexing once the file is again rotated so the complete 1 hour logs gets skipped. Before splunk resumes the indexing following error message is logged.

WatchedFile - Checksum for seekptr didn't match, will re-read entire file

Every file has a different content because each event has a timestamp so first 256 characters should not much the fishbucket.

[monitor:///data/logs/]
_TCP_ROUTING = indexers123
index = indexname
sourcetype = sourcetypename
initCrcLength = 1024
whitelist=\.log$
disabled = 0

Splunk version is 7.0.3

0 Karma

yannK
Splunk Employee
Splunk Employee

food for thoughts

  • Have you thought monitoring the rotated file too ?
    Splunk had a logic to avoid reindexing the file
    Sometimes it helps if the last events of the file were not indexed before the file rotated.

  • if the problem is during the rotation process, maybe disable the monitoring before the rotation and re-enable after.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...