Why are there differences between _time, _raw time, and indexed_time for my event?

[mprreddy51]

query:

Search to find latency:

Index=XXX source=abcd.csv | eval indexed_time=strftime(_indextime, "%+") | eval latency=_time-_indextime|table _time,indexed_time,latency,index,_raw    

Results:

_time                indexed_time                   latency   index    _raw
2015-12-18 14:01:19  Fri Dec 18 11:01:28 pst 2015   10791     XXX      2015-12-18T22:01:19Z,email,262626726,…….xss,ggggggggg

I need _raw time as a _time with out field extraction. In the above event, there are 3 different times if you see (14:01:19 ,11:01:28 pst ,22:01:19) but I need to make latency as zero.

Thanks in advance.

[esix_splunk]

Just to clarify, _indextime is the time that the event passed through the indexing queue and was written to disk on the peer. Time of this will be denoted by the local indexing peers time, not the event time itself as logged on the UF.

This implies that if your timestamp config, timezones, or event system clocks are wrong, you will have a large delta.

That being said, there are some tell-tale signs that time zone configurations are wrong versus latency due to queue fill etc. As Murali288 and Dcarmack mentions, if all events have an even time spread delta of 3hours, this means there is a time zone configuration issue that should be looked at on the ingest / forwarder tier and set to match the indexing peer.

[dcarmack_splunk]

If you don't already, you should start by adding time stamp configurations to your indexers/HWF for this particular source.

props.conf

[source::abcd.csv]
TIME_PREFIX  = ^
TIME_FORMAT = %Y-%m-%dT%T

The time stamp in the event is telling us that it was generated in UTC, 2015-12-18T22:01:19Z <-- that is what the Z means there. Your indexers appear to be in pacific time, so the events are falling on the time line correctly ( _time = UTC-8 ). If these things are both true, then the timezone offset is correct. However, the events are appearing about 3 hours in to the future. 10791 seconds of positive difference between _time-_indextime.

If this data is coming from a splunk forwarder that has a system time zone different to the indexer system time zone, then there will be time syncing issues. This does not appear to be the case here. If the forwarder is sending data that contains a time stamp from a different time zone than the forwarder system time zone, then there will be time syncing issues as well.

In past experiences, dealing with similar problems, I am going bet that the forwarder system time is UTC+3 and some application is writing a csv log in UTC. If this is true, the best way to correct this issue is to change the forwarder system time to UTC. If you can do that, then the positive latency should be corrected. Basically, your indexers think this data is coming from an alternate UTC that exists 3 hours in the future.

[Murali2888]

Hi mprreddy51,

I guess, this is due to multiple time zone issues.

Let me split up your queries.

First to make latency as zero (in an ideal scenario), the indexers timezone and the inputs props.conf timezone should be same. In your case, it seems input timezone is UTC-8:00 where as the Indexer is UTC-11:00.
But in few cases, there would be latency due to various parameters such as network delay between Indexer and Forwarder, Forwarder data rate etc.

Now to the second query, to make the _raw time as _time.
Timezone value TZ in props.conf should be set to UTC and TIME_FORMAT attribute should be configured to take the _time from input data.