I am having an issue with Splunk reindexing eStreamer log files OVER AND OVER.
12-19-2016 18:21:36.277 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:36.279 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:36.280 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:36.282 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:36.282 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:36.284 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:36.285 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:36.287 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:37.803 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:37.803 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:37.805 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:37.805 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:39.281 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:39.283 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:39.284 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:39.285 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:39.286 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:39.288 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:39.288 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:39.290 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:40.806 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:40.807 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:40.808 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:40.809 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:42.283 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:42.285 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:42.286 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:42.288 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189284'.
12-19-2016 18:21:42.288 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:42.291 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482188776'.
12-19-2016 18:21:42.291 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:42.294 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482187531'.
12-19-2016 18:21:43.810 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:43.810 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:43.812 -0500 INFO WatchedFile - Modtime is newer than stored, will reread file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:43.812 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189684'.
12-19-2016 18:21:45.286 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
12-19-2016 18:21:45.289 -0500 INFO WatchedFile - Will begin reading at offset=0 for file='/services/splunk/etc/apps/eStreamer/log/estreamer.log.1482189486'.
Has anyone ever run into this? My inputs.conf is unedited from the Cisco eStreamer for Splunk app.
# cat default/inputs.conf
# Unix/Linux inputs enabled by default
[script://./bin/client_check.py]
disabled = 0
source = eStreamer
sourcetype = client_check
index = estreamer
interval = 60
[monitor://$SPLUNK_HOME/etc/apps/eStreamer/log]
disabled = 0
source = eStreamer
sourcetype = eStreamer
index = estreamer
crcSalt = <SOURCE>
As far as I can tell, the mod times are updating...
# stat -c %x,%z,%n *
2016-12-19 18:13:07.746622111 -0500,2016-12-19 18:13:05.086528147 -0500,estreamer.log.1482188776
2016-12-19 18:18:05.669146138 -0500,2016-12-19 18:17:53.633720988 -0500,estreamer.log.1482189284
2016-12-19 18:20:24.216037498 -0500,2016-12-19 18:20:16.464764117 -0500,estreamer.log.1482189486
2016-12-19 18:39:48.360107852 -0500,2016-12-19 18:39:47.098063317 -0500,estreamer.log.1482189684
Edit 1:
I think eStreamer is keeping the file descriptor open, so mdtm and scrc never get written...
# /services/splunk/bin/splunk cmd btprobe -d /services/splunk/var/lib/splunk/fishbucket/splunk_private_db/ --file /services/splunk/etc/apps/eStreamer/log/estreamer.log.1482202447
Using logging configuration at /services/splunk/etc/log-cmdline.cfg.
key=0x40ebb995eb99a09d scrc=0x0 sptr=0 fcrc=0x8b093ad81b0a8ee3 flen=0 mdtm=0 wrtm=1482205094
# /services/splunk/bin/splunk cmd btprobe -d /services/splunk/var/lib/splunk/fishbucket/splunk_private_db/ -k 0x40ebb995eb99a09d --validate
validation result: OK
key=0x40ebb995eb99a09d scrc=0x0 sptr=0 fcrc=0x8b093ad81b0a8ee3 flen=0 mdtm=0 wrtm=1482205094
# stat -c %x,%z,%n *.log*
2016-12-20 00:00:42.377623768 -0500,2016-12-20 00:00:41.943608425 -0500,estreamer.log.1482202447
# lsof | grep eStream
splunkd 2017 splunk 38u REG 253,0 150083 924551 /services/splunk/etc/apps/eStreamer/log/estreamer.log.1482202447
... View more