Say that you have a huge volume of events, and they come in big batches. Each batch is a discrete unit, and mixing information from the most recent batch with the previous batch is unacceptable.
more givens:
Given all this, Is there a good clean way to construct a custom search or a custom view that will be sure to operate only on the events of the most recent batch?
Your best bet is to use the "head" command which can take a predicate instead of an absolute count.
For example, the following search only takes (all of) the events from the most recent second from index=_internal:
index=_internal | streamstats dc(_time) as dc_time | head dc_time==1
Your best bet is to use the "head" command which can take a predicate instead of an absolute count.
For example, the following search only takes (all of) the events from the most recent second from index=_internal:
index=_internal | streamstats dc(_time) as dc_time | head dc_time==1
This ability of head has been around since 4.1, I believe.
transaction wont work here because the set of events needs to be sliced and diced up a number of ways by a lot of different 'stats foo(bar) by baz, bat' searches, and transaction is going to put me in multivalue hell.
Wouldn't this be a good use of a transaction command? especially if you've got a well defined start and stop?
The events here are not in a single second, but this offers a tool that seems to open up a number of other ideas. Is that ability of the head command new in 4.2? It seems like I could use eval and streamstats to keep track of when I see the 'start' event and 'end' event, and then use head to terminate once I get back to the correct head event. Is that what you would do?