WatchedFile - Will begin reading at offset

[echalex]

Hi,
I have a weird issue with a bunch of files that I have recently started indexing. A whole bunch of these will end up in the index with missing events (typically missing 1 or 2 events).

For these files I'm seeing reports like:

09-27-2012 11:08:05.908 +0300 INFO  WatchedFile - Will begin reading at offset=253 for file='/path/to/file.log'

Why does the forwarder skip events like this?

[matthew_lawrenc · Accepted solution]

We've observed the same behaviour, in our case the cause was the tailing process does duplication detection with a checksum on the first and last 256 bytes of a file (so as to not index the same file twice).

We had two files with identical content being monitored.

Our resolution was to use crcSalt = A and crcSalt = B in each stanza which ensured they had differing checksums (we could do this because the duplicate files were listed in two different monitor stanzas).

You should also beware of changing the crcSalt on a running system as it will cause previously indexed files to be re-indexed.

Over a 10 minute window our splunkd.log looked like;

INFO  WatchedFile - Will begin reading at offset=1575 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=1890 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2115 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2205 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2340 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2700 for file='/tmp/logtset/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=2970 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=5490 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=6795 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=7560 for file='/tmp/logtest/test_2013101419.log'.
INFO  WatchedFile - Will begin reading at offset=9810 for file='/tmp/logtset/test_2013101419.log'.

[markbarber21]

FYI - "Do not use crcSalt = with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed."

From: http://docs.splunk.com/Documentation/Splunk/6.6.3/Data/Howlogfilerotationishandled

[_gkollias]

Thanks for this. I experiencing this as well, however my log files are not completely the same...I have something like the following:

WatchedFile - Will begin reading at offset= for file='/log/path/1/logs/file.log

WatchedFile - Will begin reading at offset= for file='/log/path/2/logs/file.log

The log paths are slightly different, but the log file names are the same. I am also already using crcSalt, but this just began 3 days ago. Still no solution found

[BenAveling]

I assume the log files are rotating? Do you know where in your log files the missing messages would have been?

[eashwar]

hey brother i got some same issue now for the past 2 days. the whole app i developed is not working. please let me know if you know the answer

[yannK]

Are you using the option ?

followTail=true

this option is misleading and for rotating files cause the offset to not start at 0, see http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

Please remove it.

[echalex]

Nope, I'm not using followTail=true, because I know - by experience - that it's not a good idea.