I would like to monitor each individual queue. The alarm can be parameterized for individual queues via lookup.
My goal is to build a dashboard in a way that each line does not only show the monitored queue but also the method (see below 3 methods) with which this queue should be monitored. How can I pack the results triggered by notifications into a dashboard using a lookup table?
The method consists of 3 notifications for each individual queue:
1) Notification for missing message flow
After the alarm is triggered, how long will no messages flow in minutes?
2) High latency notification [duration=delivery-entry] (e.g. if duration > 1200)
What is the latency in seconds until an alarm is triggered?
3) Notification of high queue level (e.g. queue_level > 100)
What is the level in number of messages, from which an alarm is triggered?
-The time until the next same alarm is triggered may be specified in minutes.
(1) Example of a queue that has not generated a message flow in the last 20 minutes:
1.1 Search:
index=hogehoge earliest=-20m@s sourcetype=syslog queue="system1"
1.2 Alerts:
Number of results is zero. Planned. Cron schedule
Also, I.e. for this queue (queue="system1" ) I should create 3 alerts (because 3 methods should be monitored for this queue) and put the results of alerts into the dashboard to make it clear. So far I haven't found a solution how to automatically create alerts for multiple queues (queue="system2" OR queue="system3" ...) and pack their results into the lookup table.
I would be very grateful for their support.
Thanks
