I have log entries from multiple hosts which contain cumulative counters. One log entry per host about every 5 minutes. Each counter on each host has a value unique to that host. Something like this:
2012-05-02T04:28:54+00:00 cp2.prod.b.example.com XACT=31568
2012-05-02T04:28:32+00:00 cp1.prod.a.example.com XACT=22622
2012-05-02T04:28:25+00:00 cp2.prod.a.example.com XACT=24623
2012-05-02T04:28:21+00:00 cp1.prod.b.example.com XACT=31140
2012-05-02T04:23:54+00:00 cp2.prod.b.example.com XACT=31500
2012-05-02T04:23:32+00:00 cp1.prod.a.example.com XACT=22600
2012-05-02T04:23:25+00:00 cp2.prod.a.example.com XACT=24600
2012-05-02T04:23:21+00:00 cp1.prod.b.example.com XACT=31100
I would like to be able to:
1) get the deltas between XACT in log entries on a per host basis
2) sum those deltas
3) graph the sum of the deltas over time (timechart)
Well, first, I'm not totally certain that you really need to compute and sum every delta -- if they're simply cumulative counters, couldn't you just subtract the first value from the last (within each time interval and for each host)? That would be more efficient.
... | bucket _time span=10min
| stats earliest(XACT) as begin
latest(XACT) as end
by _time, host
| eval delta=end-begin
| xyseries _time,host,delta
or (less preferred):
... | bucket _time span=10min
| stats earliest(XACT) as begin
latest(XACT) as end
by _time, host
| eval delta=end-begin
| timechart span=10m sum(delta) as delta by host
But for the curious, if you really wanted to do it the other way (again, the above will be faster, especially so if you have multiple indexers), you can use streamstats
:
... | streamstats current=t global=f window=2
earliest(XACT) as curr
latest(XACT) as next
by host
| eval delta=next-curr
| timechart span=10m sum(delta) as delta by host
Actually this would be a bit more RAM efficient:
... | streamstats current=t global=f window=2 range(XACT) AS delta BY host
| timechart span=10m sum(delta) AS delta BY host
The last option provided the results I needed.
Thanks for the awesome answer!
Those streamstats tips is awesome