Here is the thing:
I have 2 indexes: index_original and index_collected.
The plan is to compare/evaluate index_original events within selected timespan (say last 2 minutes) with index_collected events to see if there's any NEW EVENTS (e.g. not aleady collected to index_collected, that is) on index_original. I do not want to make one2one copy, but I want to compare with event (_time) or other unique identifier on that index if there's event available to be collected.
The target is that I have only fresh, newly collected events on index_collected - not whole stack again and then doing filtering/dedupping. Goal is to reduce index data amounts per each collection.
I have some other searches/table lookups to be done within same query, but this is the primary selector.
Any idea? Would be great help!
Will this work?
index=original OR index=collected | timechart list(_raw) as _raw by index | mvexpand _raw | where isnull(original) OR isnull(collected)
Will this work?
index=original OR index=collected | timechart list(_raw) as _raw by index | mvexpand _raw | where isnull(original) OR isnull(collected)