How to create alerts based on 2 sourcetypes and condition?

[chengka]

Hello,
I currently have alerts based on the count of services performed in the last hour. We see that < 40 indicates an issue we need to address, so I constructed a simple alert, run every hour and sends a page to support when results are less than 40.
If the count of results form (sourcetype=services History Service) < 40 per hour, alert:

 index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")

But if the server restarts, the alert condition is usually met and we get alerted which we do not want because the server restarted likely explains the lack of results.

So I want to construct a search that meets the conditions
if History < 40 and no server restart, then alert (could be stated, if the server restarted, do not alert)

sourcetype=services History Service < 40  AND 
sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start" = 0 

^ This is the restart statement

I feel like I'm overthinking this, but how do I construct a search from 2 sources, that have a conditional test like this, which can be used as an alert?

[jnudell_2]

Hi @chengka ,

You could also try the following:

index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")

| eval restart = if(sourcetype==logs_sl, 1, 0)

| stats max(restart) as restart count

| where (restart==0) AND (count < 40)

Each event will be assigned a restart value of 0, unless it's the logs_sl sourcetype that matches the restart string, which will then get assigned a restart value of 1. Then stats will find the highest value of restart, which should always be 0 unless there's a restart, in which case it's 1. Finally the where statement only shows results of there is no restart and the count is less than 40.

[somesoni2]

Give this a try

Alert search: (counting the events for each sourcetype, the count appears as a column with column name as sourcetype name)

index=stats (sourcetype=services History Service) OR (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start")
| chart count over index by sourcetype
| where 'services'<40 AND (isnull('logs_sl') OR 'logs_sl'=0)

Alert condition: Number of events greater than 0

Above search will return results if count of events from sourcetype=services History Service is less than 40 and there was no records from (sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start").

[chengka]

Thanks, that does work. I had hoped to do something like that, but I was not sure how to address the values returned by Stats. I thought it was some count prefixed name. Now I see its quite simple, just use the column name. Since I created this question, I continued to search via google and I saw a simple subsearch would work, since they are on the same host.

I know subsearches are frowned on, but the logs are small and I am only searching for 1 hour. Is my solution horrible in comparison?

index=stats (sourcetype=services History Service) NOT [search index=stats sourcetype=logs_sl "org.apache.coyote.AbstractProtocol start" | fields host]