Splunk doesn't index new file (after log rotation)

[gozulin]

Once every hour, our logfiles get copied, then the original file gets truncated and logging continues in a new file. Typical log rotation behavior.

it usually works fine, but we've noticed that sometimes, the new file doesn't get indexed. In the splunk logs, we see something like this:

04-04-2014 00:00:33.282 +0000 INFO  WatchedFile - Logfile truncated while open, original pathname file='/foo/bar/current/logs/BAR.log', will begin reading from start.

04-04-2014 00:00:33.283 +0000 INFO  BatchReader - Will retry path="/foo/bar/current/logs/BAR.log" after deferring for 10000ms, initCRC changed after being queued (before=0x3a97ce94e031dc68, after=0x691fe4ba6a203726).  File growth rate must be higher than indexing or forwarding rate.

04-04-2014 00:00:33.283 +0000 INFO  BatchReader - Removed from queue file='/foo/bar/current/logs/BAR.log'.

04-04-2014 00:00:43.211 +0000 ERROR TailingProcessor - Ignoring path="/foo/bar/current/logs/BAR.log" due to: Bug: tried to check/configure STData processing but have no pending metadata.foo

From previous readings, we've changed CHARSET from UTF-8 to AUTO without success. This is our props.conf file:

[default]
TRANSFORMS-null = setnull
CHARSET = AUTO

[foo-prod]
NO_BINARY_CHECK = 1
pulldown_type = 1

Any ideas on how to remedy this?

Thanks.

[MuS]

Hi gozulin,

take a close look at this doc about How Log File Rotation Is Handled, especially on the crcSalt part in the last chapter.

cheers, MuS

[phoenixdigital]

One other question what purpose/resolution would crcSalt = have if the log is rotated out and a brand new file of the exact name is created.

Wouldn't the crcSalt be identical?

[phoenixdigital]

Just experienced the same issue issues with a clients machine.

Logrotations have been fine for the last year or so.

Upgraded Splunk Universal Forwarder last week and got this message lastnight including the "File growth rate must be higher than indexing or forwarding rate."

Other logfiles rotated fine and continued logging to Splunk

[gozulin]

Ah, Thanks again! That is useful! will take a look 🙂

[MuS]

in addition here is a reply from Splunk Support I got in a similar case:

There is also a related bug with same error message,which required some code change which will be released through maintenance release 6.0.2, expected to be available very soon. Try that, if it won't address your problem, then get in touch with support.

[MuS]

Did you try the crcSalt = <SOURCE> option in your inputs.conf?
Also have a look at the this http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/ it has a fancy script regarding tailingProcessor

[gozulin]

Hi MuS,

Thanks for the link. I didn't see the smoking gun or potential solution in it:

256 bytes should be fine. our files have no headers, they are in the syslog format (time stamp, alert level, log msg).

The BatchReader says it will retry after 10 seconds (BatchReader - Will retry path="/foo/bar/current/logs/BAR.log" after deferring for 10000ms, initCRC changed after being queued (before=0x3a97ce94e031dc68, after=0x691fe4ba6a203726) which seems fine.

our input file specifies the exact file to be indexed, rather than folder content, so bz2 files shouldn't be an issue.

Can you elaborate?