Our Splunk Enterprise deployment has started returning inconsistent results, and I've been unable to track the source of the issue. In one example, Splunk reports that it found 34 results matching the search query, but the event viewer tab below only displays 9 of the results. I ran this same query >10 times (without any changes in search terms or time window) on our search head and received this inconsistent answer about 50% of the time. Here is a screenshot:
Additionally, some queries are returning with very large numbers of reported results but displaying no results at all in the event viewer. The large number of results is expected; the query is somewhat complex and broad. But this query structure has worked consistently for us for over a year, and suddenly it is producing these inexplicable results:
It is probably relevant to mention that we used to be using two independent search heads connected to a pool of ~10 indexers. We have recently moved to a new set of two search heads connected to a new pool of ~10 indexers - all of which is mirrored at another site, where the original equipment is being used as a replicated backup.
