We are trying to monitor an application that is busy in the day time, but is relatively quite in the night time. This is question has some root as a statistical one and how generate alerts
We are loking at traffic volumes, comparing to previous weeks at the exact same time of day, and if the traffic volumes fall off (significantly, perhaps by X number of standard deviations), then we alert.
So in the day time, we have enough events going in in our sampling period that we can generate alerts - if the traffic falls outside of normal behaviour. We happen to be looking at 8 weeks prior, building a set of reference data and if current behaviour is different (say by 3 standard deviations to previous weeks), we can alert. So far this works nicely.
But at night time, traffic volumes are very much lower. So (just as human being would do), if there is not much traffic we want to avoid the false alerts, and we stretch our sampling period (similar to in quiet periods a human being would monitor for longer) . I do not want to run this longer sampling period in the day time higher volume though, as this may make alerts slow to trigger..
Essence of the technical question though is - how do I have the same graph and apply different alert algorithms depending if I am in a 'statistical busy' period - busy periods is std dev over a 20 minute period, versus quiet periods I std dev over a longer sampling period (say 80 minutes). So a Splunk implementation question - how to change sampling period on an alert under low traffic. I would welcome any sample code..
