Hello,
I am trying to calculate the lag TIME between producers and consumers on my kafka setup. I want two know how many minutes/hours late my consumers are.
In order to do that i have two datasets consumer_offets+consume_time and producer_offset+produce_time (for each topic/consumer).
Consumer and Producer offsets are updated every minute by a script that takes a simple snapshot of offsets, so they are never really equal
I would like the find for each consumer to offset the minimum producer time min(consumer_offset>=producer_offset) so I can know what time the consumed data was produced.
I kinda managed a way around using multi-value fields, putting all producer values in a multi-value for each consumer+mvzip+mvexpand but it's limited, inefficient and ugly. Ideally, I was thinking of some sort of JOIN or Lookup using a "greater than" condition.
Anyone has an idea/thoughts how I could get the lag TIME?
thank you
