How to search and set up an alert displaying hosts that are forwarding more data compared to the average of the previous week?

[snehal8]

Hello Everyone,

I want to trigger an alert with a list of hosts that are sending more data compared to the Average of all hosts from the previous week.

Eg: The week start from Mon-Sunday has Average(divided by 7) data per host and have added 50% threshold (to compare if its increase is more than this to triggered the alert)   
 HostA: 10mb + 5 (50 %)
 HostB: 5mb   +  2.5(50%)
 HostC: 1mb  + 0.5(50%)

and Yesterday the hosts status are
 HostA: 2mb
 HostB: 4mb
 HostC: 2mb 

I have this search query which will get the hosts sending data more today

index=* earliest=-5m | eval esize=len(_raw) | stats count max(esize) by host, source | top host | fields - count

but I don't know how to write it for the above scenario.

Can any one help me on this.

Thanks

[lguinn2 · Accepted solution]

Wow - I think you are doing this the hard way. Instead of looking at the events, use the _internal index to see how much the forwarders are sending. _internal also includes the fact that the forwarders send their internal logs, but that's a pretty constant amount so you can still compute when a forwarder starts sending more than usual. Here is a search to get you started:

index=_internal source=*metrics.log group=tcpin_connections earliest=-8d
| eval Forwarder=if(isnull(hostname), sourceHost,hostname) 
| eval Today=if(_time>relative_time(now(),"@d"),"Today","PriorWeek")
| bucket span=1d _time
| stats sum(kb) as DailyKB by Forwarder Today _time
| chart avg(DailyKB)  by Forwarder Today
| where Today > PriorWeek * 1.5

This will probably run faster than your solution too, as it will not look at nearly so many events.

[somesoni2]

Amazing answer as always. Only thing here to change is the search time range. earliest=-8d would give today vs last 7 days data volume, not specifically for today vs prior week (well on monday it will be today vs prior week).

My suggestion would be to replace "earliest=-8d" with " ((earliest=@d ) OR (earliest=-1w@w1 latest=@w1))" to capture data logged for today and prior week from Mon-Sun.

[snehal8]

Thanks @somesoni will use this.

[snehal8]

Thank you @lguinn, but i am not getting mean of this "Today=if(_time>relative_time(now(),"@d"),"Today","PriorWeek")", in this "PriorWeek", is this predefined? please could you give me detail it will be grateful. thanks

[lguinn2]

The eval Today=... statement is setting a new field called Today. If the timestamp of the event is after midnight, then the value of the field is set to "Today". If the timestamp of the event is before midnight, then the field is set to "PriorWeek".

I should probably have named the field "TimeGroup" or something; Today is not a good field name, but it should work. At any rate, the chart command transforms the data so that there should be 2 columns in the results: one column named "Today" and one column named "PriorWeek".

[snehal8]

Thanks @lguinn now i understand 🙂