I have a Kafka Monitor that generates events every minute (~approx) about production and consumption rates per second.
Sample Event
{
"CONSUMER_RATE":"0.09",
"TOTAL_LOG_SIZE":"2171258",
"GROUP":"consumer_group_1",
"MSG_RATE":"0.08",
"CLUSTER":"New_York"
}
The events are generated per CLUSTER per GROUP. MSG_RATE indicates the production rate and CONSUMER_RATE indicates consumption rate.
For each CLUSTER/GROUP, I want to display the latest (most recently received events) production rate and consumption rate and color code the rows if the consumption rate is falling behind production rate by a pre-defined percentage over last X number of intervals.
If color coding is not possible, I only want to display the CLUSTER/GROUP that is failing above condition.
For displaying the latest events, I am dedup
ing on CLUSTER and GROUP. Thoughts?
Color-coding is a hassle in versions prior to 6.5 but it is built-in starting with that version. So I would first upgrade to the latest version and then you can click on the column header and program your own logic for color coding right there. Otherwise see how to do it in this app (there is an example of it).
Try a search like this:
... | dedup CLUSTER GROUP
| eval RATIO = CONSUMER_RATE / MSG_RATE
| where RATIO < YourValueHere
| table CLUSTER GROUP MSG_RATE CONSUMER_RATE RATIO
I also mentioned about the last X intervals. dedup just gives me back 1 event.
Right, but what about the rest. This answer should do it as-is. If it does not, do elaborate.
Color-coding is a hassle in versions prior to 6.5 but it is built-in starting with that version. So I would first upgrade to the latest version and then you can click on the column header and program your own logic for color coding right there. Otherwise see how to do it in this app (there is an example of it).
That answers my coloring issue. What about the query part?
Well, my biggest concern is to calculate the average for several intervals and then the color coding.