If you don't already, you should start by adding time stamp configurations to your indexers/HWF for this particular source.
props.conf
[source::abcd.csv]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%T
The time stamp in the event is telling us that it was generated in UTC, 2015-12-18T22:01:19Z <-- that is what the Z means there. Your indexers appear to be in pacific time, so the events are falling on the time line correctly ( _time = UTC-8 ). If these things are both true, then the timezone offset is correct. However, the events are appearing about 3 hours in to the future. 10791 seconds of positive difference between _time-_indextime.
If this data is coming from a splunk forwarder that has a system time zone different to the indexer system time zone, then there will be time syncing issues. This does not appear to be the case here. If the forwarder is sending data that contains a time stamp from a different time zone than the forwarder system time zone, then there will be time syncing issues as well.
In past experiences, dealing with similar problems, I am going bet that the forwarder system time is UTC+3 and some application is writing a csv log in UTC. If this is true, the best way to correct this issue is to change the forwarder system time to UTC. If you can do that, then the positive latency should be corrected. Basically, your indexers think this data is coming from an alternate UTC that exists 3 hours in the future.
... View more