Sum latest entries from multiple sources & timechart as a single line

[justingawn]

Hello,

I have multiple remote performance monitors sources, namely WMI:FOO1, WMI:FOO2 etc. up to and including WMI:FOO9. These each report on a value called BAR every 1 minute.

I am attempting to generate a timechart with a single line representing the total value of BAR across all my WMI:FOOx sources.

Additionally, any solution needs to ignore previous values of a WMI:FOOx source if that source has not reported a value for BAR within a set time period, say 2 minutes (in case the server goes down, I don't want the last value being included in a sum.)

Thanks in advance,
Justin

[mannioke]

Well this isn't perfect to your case but could be a useful substitution, if you are will to accept the modifications in behavior. I have a very similar situation with multiple QoS across multiple Interfaces across multiple Hosts. Because some interfaces utilization is drastically different than another I want to watch per host, not per interface per host. Thus far any indication of no response appears to be reporting as a gap in my chart.

My goal was to measure the interval change (new - old = delta), my only problem now is because of the forced values my first interval has an over inflated value that I need to drop. A sub search hasn't worked thus far, but this is a work in progress.

The forced values were required as a result of any of the QoS values could be null and that would result in a null sum regardless if the other 8 QoS contained values.

index=bulkstats (host=host-01 OR host=host-02) QoS8_dwlinkpkt_drop=* QoS8_uplinkpkt_drop=*
| reverse
| streamstats global=f current=f
last(QoS1_dwlinkpkt_drop) as p_QoS1_dwlinkpkt_drop, last(QoS1_uplinkpkt_drop) as p_QoS1_uplinkpkt_drop,
last(QoS2_dwlinkpkt_drop) as p_QoS2_dwlinkpkt_drop, last(QoS2_uplinkpkt_drop) as p_QoS2_uplinkpkt_drop,
last(QoS3_dwlinkpkt_drop) as p_QoS3_dwlinkpkt_drop, last(QoS3_uplinkpkt_drop) as p_QoS3_uplinkpkt_drop,
last(QoS4_dwlinkpkt_drop) as p_QoS4_dwlinkpkt_drop, last(QoS4_uplinkpkt_drop) as p_QoS4_uplinkpkt_drop,
last(QoS5_dwlinkpkt_drop) as p_QoS5_dwlinkpkt_drop, last(QoS5_uplinkpkt_drop) as p_QoS5_uplinkpkt_drop,
last(QoS6_dwlinkpkt_drop) as p_QoS6_dwlinkpkt_drop, last(QoS6_uplinkpkt_drop) as p_QoS6_uplinkpkt_drop,
last(QoS7_dwlinkpkt_drop) as p_QoS7_dwlinkpkt_drop, last(QoS7_uplinkpkt_drop) as p_QoS7_uplinkpkt_drop,
last(QoS8_dwlinkpkt_drop) as p_QoS8_dwlinkpkt_drop, last(QoS8_uplinkpkt_drop) as p_QoS8_uplinkpkt_drop,
last(QoS9_dwlinkpkt_drop) as p_QoS9_dwlinkpkt_drop, last(QoS9_uplinkpkt_drop) as p_QoS9_uplinkpkt_drop by group, host
| chart sum(eval((if(isnotnull(QoS1_dwlinkpkt_drop),QoS1_dwlinkpkt_drop,0) + if(isnotnull(QoS2_dwlinkpkt_drop),QoS2_dwlinkpkt_drop,0) + if(isnotnull(QoS3_dwlinkpkt_drop),QoS3_dwlinkpkt_drop,0) +
if(isnotnull(QoS4_dwlinkpkt_drop),QoS4_dwlinkpkt_drop,0) + if(isnotnull(QoS5_dwlinkpkt_drop),QoS5_dwlinkpkt_drop,0) + if(isnotnull(QoS6_dwlinkpkt_drop),QoS6_dwlinkpkt_drop,0) +
if(isnotnull(QoS7_dwlinkpkt_drop),QoS7_dwlinkpkt_drop,0) + if(isnotnull(QoS8_dwlinkpkt_drop),QoS8_dwlinkpkt_drop,0) + if(isnotnull(QoS9_dwlinkpkt_drop),QoS9_dwlinkpkt_drop,0)) -
(if(isnotnull(p_QoS1_dwlinkpkt_drop),p_QoS1_dwlinkpkt_drop,0) + if(isnotnull(p_QoS2_dwlinkpkt_drop),p_QoS2_dwlinkpkt_drop,0) + if(isnotnull(p_QoS3_dwlinkpkt_drop),p_QoS3_dwlinkpkt_drop,0) +
if(isnotnull(p_QoS4_dwlinkpkt_drop),p_QoS4_dwlinkpkt_drop,0) + if(isnotnull(p_QoS5_dwlinkpkt_drop),p_QoS5_dwlinkpkt_drop,0) + if(isnotnull(p_QoS6_dwlinkpkt_drop),p_QoS6_dwlinkpkt_drop,0) +
if(isnotnull(p_QoS7_dwlinkpkt_drop),p_QoS7_dwlinkpkt_drop,0) + if(isnotnull(p_QoS8_dwlinkpkt_drop),p_QoS8_dwlinkpkt_drop,0) + if(isnotnull(p_QoS9_dwlinkpkt_drop),p_QoS9_dwlinkpkt_drop,0)))) as DW-Link,
sum(eval((if(isnotnull(QoS1_uplinkpkt_drop),QoS1_uplinkpkt_drop,0) + if(isnotnull(QoS2_uplinkpkt_drop),QoS2_uplinkpkt_drop,0) + if(isnotnull(QoS3_uplinkpkt_drop),QoS3_uplinkpkt_drop,0) +
if(isnotnull(QoS4_uplinkpkt_drop),QoS4_uplinkpkt_drop,0) + if(isnotnull(QoS5_uplinkpkt_drop),QoS5_uplinkpkt_drop,0) + if(isnotnull(QoS6_uplinkpkt_drop),QoS6_uplinkpkt_drop,0) +
if(isnotnull(QoS7_uplinkpkt_drop),QoS7_uplinkpkt_drop,0) + if(isnotnull(QoS8_uplinkpkt_drop),QoS8_uplinkpkt_drop,0) + if(isnotnull(QoS9_uplinkpkt_drop),QoS9_uplinkpkt_drop,0)) -
(if(isnotnull(p_QoS1_uplinkpkt_drop),p_QoS1_uplinkpkt_drop,0) + if(isnotnull(p_QoS2_uplinkpkt_drop),p_QoS2_uplinkpkt_drop,0) + if(isnotnull(p_QoS3_uplinkpkt_drop),p_QoS3_uplinkpkt_drop,0) +
if(isnotnull(p_QoS4_uplinkpkt_drop),p_QoS4_uplinkpkt_drop,0) + if(isnotnull(p_QoS5_uplinkpkt_drop),p_QoS5_uplinkpkt_drop,0) + if(isnotnull(p_QoS6_uplinkpkt_drop),p_QoS6_uplinkpkt_drop,0) +
if(isnotnull(p_QoS7_uplinkpkt_drop),p_QoS7_uplinkpkt_drop,0) + if(isnotnull(p_QoS8_uplinkpkt_drop),p_QoS8_uplinkpkt_drop,0) + if(isnotnull(p_QoS9_uplinkpkt_drop),p_QoS9_uplinkpkt_drop,0)))) as UP-Link by _time, host

[jtrucks]

First, to get results from multiple sources, use this in your search:

source=WMI:FOO*

So, something like:

source=WMI:FOO* BAR=* | timechart span=1m sum(BAR) useother=false

Could you clarify (in comments or by editing the question) what specifically you are trying to ignore? You can specify stuff like NOT BAR=0 or things like that...

[justingawn]

Regarding the ignoring of values - if the solution to above does involve summing of last BAR values of each WMI:FOOx source, I would want to ensure that in the event of a server being unreachable, that its last BAR value is not summed into the graphed value.

[justingawn]

I had tried something similar, a sum with a span equal to the poll interval.

Occasionally a server would respond immediately resulting in 2 results for the same source within the 60s span. Conversely sometimes a server would take a while to respond meaning that there would be no value for the source counted within the span.

This resulted in blips of +/- a single BAR value on the graph. Tweaking the span would either decrease duplicate source counts and increase missed counts, or vice versa.

Can a sum be done on the last values for each WMI:FOOx source, rather than rely on a time based span?