_time vs _indextime: Why is it necessary to sync the two timestamps, and which one is considered using the time range picker?

[abhayneilam]

Hi,

I have an understanding that _time --> is the event time (the time which is present in the event means the time when the event was generated) and "_indextime" --> is the index time (the time when the events have been indexed).
Now, if my event time and indexed time is not same for some reason (let's say, if my forwarder is down for an hour, then the event time and the index time will have a 1 hour diff), can I consider those events as a bad events whose index-time and event time is not same?

And also, why it is necessary to always sync event time and index-time? What is the benefit of syncing these two times, and when we click on the time using "Timepicker", which time does it consider?

Please help !!

[scelikok]

Hi @chaitali_1994,

There are too many possibilities if the latency is bigger than a few seconds, some of them are below;  

- Client machines may be offline or cannot send logs to Splunk for sometime, think about an employees are using the  laptop at home, their logs will arrive Splunk next business day. You will see huge latency.

- The source time maybe wrong, not using NTP.

- Timestamp extraction problem

- Timezone problem, for example source is sending timestamps in UTC but timestamp has no sign.

[scelikok]

Hi @chaitali_1994,

You can use below query;

| your_search
| eval indextime=_indextime
| eval latency=indextime-_time

[chaitali_1994]

Hi @scelikok,

Thanks for providing the query to check the latency. We have applied the similar query and got to know about the latency.

Could you please help us on a solution to fix the latency issue and what could be the possibilities for the latency?

Much appreciated your help!!

[esix_splunk]

The difference between _time and _indextime helps us understand when the events are seen, vs when the disk is written to disk on the actual indexers.
What having this enables us to do, is understand latency between ingest time (event timestamp) and when this is written to disk. There should be nominal differences in these unless this is expected (such as with batch read.)
Where we start seeing huge deltas between these, is usually indicative of performance issues at the parsing / indexing layer and we recommend scaling out the indexing layer.

[chaitali_1994]

@esix_splunk Could you please help us understand more in details? Also please tell how can we check the latency between indextime and _time?

[martin_mueller]

The time range picker works on the event time in the _time field.

It's not strictly necessary to sync up index time with event time, it's just an indication of indexing delay or old data. Whether that's bad or not depends on your circumstances. For example, if some source only delivers a batch of data for yesterday every night then seeing up to a day of difference between index time and event time is unavoidable. I wouldn't call those events bad, just delayed.