Why are my events not in time order?

[iqtroy]

We just upgraded our Splunk server to version 7.0. I created a query that has a time range Between 05/19/2018 04:28:00.000 and 05/19/2018 08:47:00.000. I list 50 events per page. I navigate through pages and I see events in random order. On page 17 (page with oldest events) I see events with these times in this order:
5/19/18 6:11:09.115 AM
5/19/18 5:35:07.463 AM
5/19/18 5:31:00.510 AM
5/19/18 6:08:27.757 AM
5/19/18 6:08:27.753 AM
5/19/18 5:31:00.510 AM
and so on....

There are 2 problems, 1 is that they are not in expected order and 2 the oldest events should have a time close to 05/19/2018 04:28:00.000.

What is going on here?

[mhoogcarspel_sp]

This reads like SPL-154973 actually, fixed in 7.1.3+
http://docs.splunk.com/Documentation/Splunk/7.1.3/ReleaseNotes/Fixedissues

Upgrade SH and IDX to 7.1.4+ (can't recommend to upgrade to 7.1.3 for other issues).

[seshi]

Hi chanfoli, we have a clustered deployment with a single search head and recently upgraded to 7.1.0.
* single SH with distributed search enabled
* clustered indexers

[chanfoli]

Thanks for the reply. I also found a question alluding to similar symptoms from another customer from the beginning of the month using the 7.1.x tag - https://answers.splunk.com/answers/655529/search-returning-duplicatedwrong-results-after-upg.html

[iqtroy]

Yes, seshi answered for me. I thought we had version 7.0 but seshi did the upgrade so he knows best.

[chanfoli]

I have a support case open with what sounds like similar behavior in 7.1.0 - This is with a SH and Indexer cluster, we also notice more strangeness when selecting time ranges on the timeline, it does not properly bound the earliest and latest events and sometimes duplicate events are seen. I am curious about your deployment type, i.e. is it a SH cluster or single SH Are you searching against in indexer cluster or single indexer, and if it is a cluster is it mutli-site?