Is the file structured data? If not, you probably have INDEXED_EXTRACTIONS defined for this source, sourcetype, or globally; this will cause issues with the crc. While the issue manifests itself as TailingProcessor errors, btprobe behaves in the same way and is an easy test. You can replicate this with the license_usage.log, here with the setting sitting outside of any stanza (not recommended!):
> head -1 ./splunk/etc/system/local/props.conf
INDEXED_EXTRACTIONS = CSV
> ./splunk/bin/splunk cmd btprobe -d ./splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file ./splunk/var/log/splunk/license_usage.log
Using logging configuration at ./splunk/etc/log-cmdline.cfg.
terminate called after throwing an instance of 'FileContentException'
what(): Ran out of data while looking for end of header
Aborted (core dumped)
btprobe crashes!
We don't index license_usage.log!
> ./splunk/bin/splunk search 'index=_internal | stats count by source'
source count
----------------------------------------------------------------------- -----
./splunk/etc/splunk.version 1
./splunk/var/log/splunk/conf.log 5
./splunk/var/log/splunk/first_install.log 4
./splunk/var/log/splunk/metrics.log 68079
./splunk/var/log/splunk/migration.log.2016-02-18.19-59-41 12
./splunk/var/log/splunk/mongod.log 412
./splunk/var/log/splunk/scheduler.log 48
./splunk/var/log/splunk/splunkd-utility.log 65
./splunk/var/log/splunk/splunkd.log 1837
./splunk/var/log/splunk/splunkd_access.log 166
./splunk/var/log/splunk/splunkd_stderr.log 10
./splunk/var/log/splunk/splunkd_ui_access.log 2363
./splunk/var/log/splunk/web_access.log 131
./splunk/var/log/splunk/web_service.log 1101
If we rectify this, btprobe works, and we index the log:
> sed -i 's/^INDEXED_EXTRACTIONS/#INDEXED_EXTRACTIONS/g' ./splunk/etc/system/local/props.conf
> ./splunk/bin/splunk restart &>/dev/null
> ./splunk/bin/splunk search 'index=_internal | stats count by source'
source count
----------------------------------------------------------------------- -----
./splunk/etc/splunk.version 2
./splunk/var/log/splunk/conf.log 6
./splunk/var/log/splunk/first_install.log 4
./splunk/var/log/splunk/license_usage.log 11877
./splunk/var/log/splunk/metrics.log 69386
./splunk/var/log/splunk/migration.log.2016-02-18.19-59-41 13
./splunk/var/log/splunk/mongod.log 631
./splunk/var/log/splunk/scheduler.log 66
./splunk/var/log/splunk/splunkd-utility.log 107
./splunk/var/log/splunk/splunkd.log 3100
./splunk/var/log/splunk/splunkd_access.log 214
./splunk/var/log/splunk/splunkd_stderr.log 17
./splunk/var/log/splunk/splunkd_ui_access.log 2366
./splunk/var/log/splunk/web_access.log 134
./splunk/var/log/splunk/web_service.log 1835
> ./splunk/bin/splunk cmd btprobe -d ./splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file ./splunk/var/log/splunk/license_usage.log
Using logging configuration at ./splunk/etc/log-cmdline.cfg.
key=0x96c781d205619afe scrc=0xf665afd1903bda74 sptr=4720849 fcrc=0xa3bd152f008960fc flen=0 mdtm=1455235563 wrtm=1455236118
... View more
