Alert if number of events *by any one host* rises by 100%

[malbery]

Hi,

If I limit by search to a specific host, then I know how to trigger an alert if the number of events for that saved search rises by 100%. Naturally I don't want to create a saved search for each host that I have. Is there a way I can alert for this with a single saved search?

Cheers,
Merlyn

[rtadams89]

If you are currently running your search every hour, looking at the event counts in the last hour, and comparing that to an hour before, you could:

Run your search every hour, looking over the last TWO hours, then use a combination of buckets and stats to get the count of events by host, for the first hour and the second hour. You then have a table (you don't have to litterally "| table" it) of each host, the events for that host between two hours ago and one hour ago, and the events for that host between one hour ago and now. At the end of your search, string, use | where eventCount2 > 2 * eventCount1

This will then return you a list of all hosts that had twice as many events in the last hour, as the hour before. Set the alert to trigger if the number of results is > 0. You will then be alerted anytime one or more hosts has 100% more events this hour than last hour, and in the alert will be provided a list of which hosts met that criteria.

[kristian_kolb]

Well, I don't really know what you are planning to do, but the following search will list all hosts which have sent a 100% more messages in the last minute, than compared with a minute half an hour ago.

    * earliest =-30m latest=-29m 
| stats count AS then_events by host 
| join host type=outer 
[search earliest=-1m latest=now 
| stats count AS now_events by host] | fillnull 
| eval change_in_percent=round((now_events/then_events -1) * 100,1) 
| where change_in_percent > 100

Unfortunately this uses the count of events as indexed, not information from a summary or internal index, which means that it will have to go through a lot of data if you're looking at longer time spans. Also it uses join, which is expensive.

You could give it a try to see if it works for you. Should not require any modifications just for a test run.

Hope this helps,

Kristian

[_d_ · Accepted solution]

Hi Malbery,

This is is not possible with the current version of Splunk. The next major release (4.3) will have a feature that does exactly that - it will be called per result alerting.

> please upvote and accept answer if you find it useful - thanks!