Alerting

setup alert based on host event count compared with other host

prakashaig
Explorer

I have 3 webservers which takes the traffic and that is load balanced with least connection based without any sticky sessions, so the traffic will be evenly loaded b/w these servers. looking to create alert if any of the host have less event count comparatively. have the below basic query which will look for specific event on all 3 access logs. we can get alert if there is no event by adding | search eventCount=0 but i need to get alert comparing to other host for example x server has 25 events and other server has 100 events which is above my threshold (75% difference). this will help me trouble shot the LB or may the process is X server is taking longer time to respond or something.

index=x AND (host="x" OR host="y" OR host="z" ) AND source="*access" AND "xyz.com"
| search ResponseCode=200
| inputlookup append=t apache_httpd.csv
| stats count as eventCount by host

apache_httpd.csv is nothing but as below
host
x
y
z

Tags (1)
0 Karma

renjith_nair
Legend

@prakashaig ,

You might need to find an avg event count (baseline) for the hosts and calculate the percentage of difference based on that.

Try if this works for you

your current search
|eventstats avg(eventCount ) as Avg
|eval percentage=abs(round((eventCount-avg)/avg*100,2))

Alert based on the percentage of deviation

---
What goes around comes around. If it helps, hit it with Karma 🙂
0 Karma
Get Updates on the Splunk Community!

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...

Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework ...

WATCH NOWAs AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting ...