Alerting

How to set up an alert if the same error occurs more than 2 times in one hour per server?

dolejh76
Communicator

I saw that someone asked something similar before but it was in reference to different data and I couldn't get it to work. Right now I am pulling all the data from windows servers. I would like to set up an alert if the same error (or warning) occurs more than twice in one hour per server. obvious this would not be for informational items etc.

This would run every hour

So for example

Host 1 - error1234@13:04 error1234@13:10 error1234@13:14
Host 2 - error1234@13:04

Alert sent at 14:00 - Host 1 has had error1234 occur x times from 13:00 to 14:00

Appreciate the help - thanks!

Tags (2)
0 Karma
1 Solution

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

View solution in original post

MuS
Legend

Hi dolejh76,

Try something like this:

Your base search here AND error1234 | bucket _time span=1h | stats count by host | where count>2

Hope this helps...

cheers, MuS

dolejh76
Communicator

Ok I manage to make forward progress...

index="wineventlog" AND Type=Error OR Type=Warning| bucket _time span=1h | stats count by host | where count>2

This results in a list of hosts and the count of messages over 2 times per hour.

How would I go about getting the output to be something like...

Host1
Error1234 - 10x
Error2345 - 7x
Warning1234 - 10x

Host1
Error1234 - 3x
Error2345 - 7x
Warning1234 - 6x

etc

The end goal will be to send these alerts to the helpdesk so that they see that there are continuous errors that are occurring frequently or warnings on a server so that we look at the root cause and resolve.

Thanks
John

0 Karma

dolejh76
Communicator

index="wineventlog" AND Type=Error OR Type=Warning | bucket _time span=1h | stats count by host,Message | where count>2

Think this gives me what I want. At least gives me a lot to look at...

John

0 Karma

MuS
Legend

Glad to see you got what you want 🙂

0 Karma

dolejh76
Communicator

How do I specify type error and type warning?

0 Karma

dolejh76
Communicator

I guess I should clarify - any error, not just 1234... - so do I just remove "AND error1234" from your string?

Thanks

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...