So far I've been dissatisfied with the various volume-used searches I've tried. My latest attempt includes not only the volume in the last full hour, but also the % change from the previous hour, as well as the busiest source host and its volume.
index=_internal group="per_index_thruput" NOT series="_*" NOT series="history" NOT series="summary" earliest=-2h@h latest=@h | eval mb=kb/1024| timechart span=1h sum(mb) by series | delta main as hourly_change | eval perc_change=hourly_change/(main-hourly_change) | fields + perc_change,main | tail 1 | appendcols [search earliest=-1h@h latest=@h | eval KB=length(_raw)/1024 | stats sum(eval(KB/1024)) as MB by host | sort -MB | head 1 ]
Then I alert on a where clause for total MB and/or percentage change hour-over-hour in an attempt to catch big jumps in volume or rates that, if sustained, would push our licensed limit.
What's your preferred method of tracking/alerting on volume?
