What is _indextime, really?

[gsumner]

We have had several examples recently where scheduled searches appear to run in the _internal log, complete successfully, and find no results when results were available. Using _indextime vs _time to determine the lag on the results it seems like there is no reason why the results were not found by the search.

We have an open case with support but this makes me wonder:

Where does _indextime fall in the indexing process? Is it when the event is written to the first bucket? Is it when the first searchable copy is written? Is it when the full replication/search factors have been met across the cluster (unlikely)? As you can probably guess I'm wondering if we can get into a state where data is written/indexed but not actually searchable.

For reference, we have a multisite peer cluster with 8 members over 2 sites (2 AZs in the same AWS region).

Thanks!

[jnicholsenernoc]

I came up with a search that shows _indextime almost certainly does not mean "available for search".

The search uses the addinfo command to get the time when the search was executed, then compares that time to the most recent available event's _indextime as well as the most recent event's event time (_time).

Here is the search to tack on to the end of some base search you are interested in:

YOUR_SEARCH_HERE
| eval lag_sec=_indextime-_time |
stats max(lag_sec) as max_lag max(_indextime) as max_index_time max(_time) as max_event_time 
| addinfo 
| eval index_lag_for_search = info_search_time - max_index_time
| eval event_lag_for_search = info_search_time - max_event_time
| table info_search_time info_min_time info_max_time, max_event_time, max_index_time, max_lag, index_lag_for_search, event_lag_for_search

For the timeframe, set it to be small and recent, earliest = -1m and latest = now, so you are sure to get the most recent event data available. The base search should be against a high frequency data source (in my case the events are written at least once per second to the index involved).

For my base search, I found that the maximum indexing lag never exceeds 2-3 seconds for any of the events in the index, but the "event_lag_for_search" measurement, which shows the most recent event available for the search can sometimes fall behind by 30+ seconds.

@gsumner , it seems like this can help answer at least one of your questions:
Is it when the first searchable copy is written? NO, _indextime does not seem to mean that the event is searchable/available for search

[ddrillic]

https://answers.splunk.com/answers/1001/how-do-i-determine-when-an-event-was-indexed.html speaks about it -
"Since Splunk 4.0, the indexing machine will add an index time field called _indextime to events as they are written to disk."
.. | eval indextime = _indextime

You can also run the following -
eval latency=_time-_indextime|

[gyslainlatsa]

hi gsumner,

The `_indextime` field contains the time that an event was indexed, expressed in Unix time. You might use this field to focus on or filter out events that were indexed within a specific range of time.

[somesoni2]

The _indextime is a default field available, it's time when Splunk writes the event to disk OR about to write to disk. Again not all data written is searchable immediately (on the indexer where it's stored as primary bucket + additional indexers based on search factor), so I can't think of a period when data is in Splunk but not searchable.

[gsumner]

Thanks somsoni2. Can you explain a little more why you think there wouldn't be a time when data is in Splunk but not searchable? As you said, the data is "indexed" when it is written to disk, but that doesn't make it immediately searchable. And it certainly could be written to the primary/origin indexer but not replicated to other sites.

[somesoni2]

IMO, when an Indexer receives the data (not replication) and when it's being processed, both the raw data and tsidx/metadata info (which makes the data bucket searchable) is generated and both are written to disk. So, any data which is indexed in a primary bucket is searchable immediately. In case of replicated buckets howevers, only the replicated copied equal to search factor are make searchable on other indexers, but again there always be a primary bucket for every data somewhere in one of the indexer and it's searchable (provided all indexers are up).

[jnicholsenernoc]

The search I posted in my answer below seems to show that the _indextime field doesn't mean that the particular event is available for search. For a couple of our base searches, the _indextime is never more than 2-3 seconds after the event time, but the most recent event available to a particular search can be 30+ seconds old.

[gsumner]

OK thanks, that makes sense (that _indextime represents both raw data and metadata being written to the primary bucket).

Thanks!