There are continuous transactions' log into Splunk.
Is it possible to let Splunk alert when some transaction's duration is more than 10-times the average duration?
For example, average duration is A, and if some transaction's duration is over 10A, then Splunk raises an alert.
I tried "
Any one can give some advice? Thanks!
stats will transform the event rows themselves. However eventstats
will leave the event rows alone, and it will add the stats rows onto every event.
<some search=""> | eventstats avg(duration) as avgdur | search duration > avgdur*10
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/eventstats
while on the topic you might look at streamstats
which is similar but where eventstats
will run statistics over the entire resultset, streamstats
will for each row only compute the stats up to that row.
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/streamstats
The two searches will behave very differently! streamstats will only calculate the average from the earliest point in the set up to the current row. And then each row might end up with a slightly different average. eventstats on the other hand will compute the average over the entire set, and then paint that single average onto every row.
Of course, the real trick here is to be confident that 2.5 (or whatever the number you choose) is the right one. Set it too high and you'll miss potentially interesting spikes. Set it too low and you get bombarded.
I also couldn't get your example (using eventstats) to work on real data where I knew I had a spike that was about 2.5x larger than the average...
However, I could get it to work using streamstats:
(had to use "where", not "search", by the way)
You're absolutely right. Thanks. I'll fix the typo in the answer.
Is there a typo in your answer? Should it really be:
??