Hello,
I currently have alerts based on the count of services performed in the last hour. We see that < 40 indicates an issue we need to address, so I constructed a simple alert, run every hour and sends a page to support when results are less than 40.
If the count of results form (sourcetype=services History Service) < 40 per hour, alert:
index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")
But if the server restarts, the alert condition is usually met and we get alerted which we do not want because the server restarted likely explains the lack of results.
So I want to construct a search that meets the conditions
if History < 40 and no server restart, then alert (could be stated, if the server restarted, do not alert)
sourcetype=services History Service < 40 AND
sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start" = 0
^ This is the restart statement
I feel like I'm overthinking this, but how do I construct a search from 2 sources, that have a conditional test like this, which can be used as an alert?
Hi @chengka ,
You could also try the following:
index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")
| eval restart = if(sourcetype==logs_sl, 1, 0)
| stats max(restart) as restart count
| where (restart==0) AND (count < 40)
Give this a try
Alert search: (counting the events for each sourcetype, the count appears as a column with column name as sourcetype name)
index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")
| chart count over index by sourcetype
| where 'services'<40 AND (isnull('logs_sl') OR 'logs_sl'=0)
Alert condition: Number of events greater than 0
Above search will return results if count of events from sourcetype=services History Service
is less than 40 and there was no records from (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")
.
Thanks, that does work. I had hoped to do something like that, but I was not sure how to address the values returned by Stats. I thought it was some count prefixed name. Now I see its quite simple, just use the column name. Since I created this question, I continued to search via google and I saw a simple subsearch would work, since they are on the same host.
I know subsearches are frowned on, but the logs are small and I am only searching for 1 hour. Is my solution horrible in comparison?
index=stats (sourcetype=services History Service) NOT [search index=stats sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start" | fields host]