I'm taking iis logs from an Exchange server via a forwarder on that system. Originally I had TZ = GMT on the etc/system/local/props.conf file for this sourcetype. But I have been having a problem with not being able to view events in real-time, so I thought maybe if the TZ was set on the forwarder's props.conf via the deployment app, that would fix that issue. I made the change, and after the reload deploy-server took I restarted splunkd to flush the TZ setting out which was removed on the indexer. This did nothing to help the issue, so I put everything back the way that it was before, again with a reload and for the indexer a splunkd restart. I received an email alert a short time later about this index (volume alert I wrote). Based on the size of the log file, looks like splunk reindexed the file twice. Why would changing the time zone cause that? I thought reindexing would happen only if the CRC for the beginning of the file changed. It is oky now, but just very bizarre...
I do not know for sure but my theory is that if the time zone changes then splunk sees the time of the file's modtime differently, and that causes the reindex. What made me think of this is this comment in the inputs.conf documentation:
alwaysOpenFile = 0 | 1
If set to 1, Splunk opens a file to check if it has already been indexed.
Only useful for files that don't update modtime.
Obviously, modtime plays a role in opening (and perhaps indexing or reindexing) or it would not be mentioned here. But in any case, whether this is a correct theory or not is moot, for I have the proof that it happens.