We have high cardinality data -- virtually every event is unique except for a small percentage of cases that we care about. So we're finding that we have count the unique ids, track them somehow in order to find the duplicates. Its just not feasible in Splunk when we have millions of events per minute.
Example:
search | stats count by unique_id | where count>1
(Millions of events per minute, results in a few hundred events where count>1). Summary indexing is not really a solution here since the unique_id could cross time/minute boundary.
|fields unique_id
|stats list( unique_id) AS id
| where mvfilter(mvcount(id)>1) != NULL
May be setup indexed time field extraction for your unique_id field so you could use tstats with it.