Alert on anomalous growth of thruput per sourcetype

[IgorB]

I want to create a scheduled search that will be able to trigger an alert if a there's a sourcetype that has a thruput much higher than the average for that sourcetype for a given time period.

Rational: be able eliminate license violations by identifying "peaking" sourcetypes in a timely manner

It's quite easy to see the peaks on a timechart, e.g.

index=_internal group=per_sourcetype_thruput | timechart per_second(kb) by series

[gkanapathy]

Depending exactly what you're trying to do, there are many ways to do this. The simplest is to use trendline/streamstats:

index=_internal group=per_sourcetype_thruput 
| bucket _time span=5m
| stats sum(kb) as kb by _time,series
| makecontinuous _time span=5m 
| streamstats global=f current=f window=24
    avg(kb) as moving_avg_kb
    stdev(kb) as stdev_kb
  by series

And then alert conditionally:

| where kb > (moving_avg_kb+(0.5*stdev_kb))

Of course, there is much tuning on what you consider to be "allowable" peaks and over what period of time. (e.g., the above will not catch a source growing slowly over a two-hour or longer period) More complex requirements are going to require either more searches or much more complex ones.