Alerting

How to set up an alert for unresponsive hosts with different time allowances for different sourcetypes?

glenrattay
Engager

I have an alert that will trigger if a host does not respond for 60 minuets. I would like to be able to be able to specify sourcetypes so that we can have different allowances. e.g 1 day, 6 hours and 60 minuets.

| metadata type=hosts | eval responsive=if(recentTime < now() - 60*60*3, "no", "yes") | eval defunct=if(recentTime > now() - 60*60*24*3, "no", "yes") | where responsive="no" | convert ctime(recentTime) as last_contact | fields host, responsive, defunct, last_contact | rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

Any advice on how I can achieve this?

0 Karma

lguinn2
Legend

Well, your first problem is that the metadata command does not return any information about sourcetypes when you specify the type of hosts. I guess you could do it like this, assuming that 60 minutes is the minimum time for all sourcetypes:

index=* [ | metadata type=hosts | where recentTime < (now() - 3600) | fields host ]
| stats latest(_time) as lastTime by host sourcetype
| eval responsive = case(sourcetype="A" AND lastTime < (now() - 3600),"no",
                  sourcetype="B" AND lastTime < (now() - 86400),"no",
                  etc.,
                 1==1,"yes")
| where responsive="no" 
| eval defunct=if(lastTime > now() - 60*60*24*3, "no", "yes") 
| eval last_contact = strftime(lastTime,"%m/%d/%Y %X")
| sort host sourcetype
| table host, sourcetype, responsive, defunct, last_contact 
| rename host as "Host" responsive as "Responsive" defunct as "Defunct" last_contact as "Last Update"

This is going to be a lot slower than your original search. You could set up a lookup table that contains the sourcetypes and the time allowance for each sourcetype. That will eliminate the complex case statement, but it won't make the search any faster.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...