Once every hour, our logfiles get copied, then the original file gets truncated and logging continues in a new file. Typical log rotation behavior.
it usually works fine, but we've noticed that sometimes, the new file doesn't get indexed. In the splunk logs, we see something like this:
04-04-2014 00:00:33.282 +0000 INFO WatchedFile - Logfile truncated while open, original pathname file='/foo/bar/current/logs/BAR.log', will begin reading from start.
04-04-2014 00:00:33.283 +0000 INFO BatchReader - Will retry path="/foo/bar/current/logs/BAR.log" after deferring for 10000ms, initCRC changed after being queued (before=0x3a97ce94e031dc68, after=0x691fe4ba6a203726). File growth rate must be higher than indexing or forwarding rate.
04-04-2014 00:00:33.283 +0000 INFO BatchReader - Removed from queue file='/foo/bar/current/logs/BAR.log'.
04-04-2014 00:00:43.211 +0000 ERROR TailingProcessor - Ignoring path="/foo/bar/current/logs/BAR.log" due to: Bug: tried to check/configure STData processing but have no pending metadata.foo
From previous readings, we've changed CHARSET from UTF-8 to AUTO without success. This is our props.conf file:
[default]
TRANSFORMS-null = setnull
CHARSET = AUTO
[foo-prod]
NO_BINARY_CHECK = 1
pulldown_type = 1
Any ideas on how to remedy this?
Thanks.
Hi gozulin,
take a close look at this doc about How Log File Rotation Is Handled, especially on the crcSalt
part in the last chapter.
cheers, MuS
One other question what purpose/resolution would crcSalt =
Wouldn't the crcSalt be identical?
Just experienced the same issue issues with a clients machine.
Logrotations have been fine for the last year or so.
Upgraded Splunk Universal Forwarder last week and got this message lastnight including the "File growth rate must be higher than indexing or forwarding rate."
Other logfiles rotated fine and continued logging to Splunk
Ah, Thanks again! That is useful! will take a look 🙂
in addition here is a reply from Splunk Support I got in a similar case:
There is also a related bug with same error message,which required some code change which will be released through maintenance release 6.0.2, expected to be available very soon. Try that, if it won't address your problem, then get in touch with support.
Did you try the crcSalt = <SOURCE>
option in your inputs.conf?
Also have a look at the this http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/ it has a fancy script regarding tailingProcessor
Hi MuS,
Thanks for the link. I didn't see the smoking gun or potential solution in it:
256 bytes should be fine. our files have no headers, they are in the syslog format (time stamp, alert level, log msg).
The BatchReader says it will retry after 10 seconds (BatchReader - Will retry path="/foo/bar/current/logs/BAR.log" after deferring for 10000ms, initCRC changed after being queued (before=0x3a97ce94e031dc68, after=0x691fe4ba6a203726) which seems fine.
our input file specifies the exact file to be indexed, rather than folder content, so bz2 files shouldn't be an issue.
Can you elaborate?