Alerting

How to create an alert if no data is generated from a host?

ashari
Explorer

I want to run a search in splunk to find out that all the devices attached to the splunk server are generating logs. If I dont hear from a device I should receive a alert.

thanks

Tags (1)

martin_mueller
SplunkTrust
SplunkTrust

Here's a quick way to check for hosts not having sent data in a while:

| tstats latest(_time) as latest where index=* by host | where latest < relative_time(now(), "-1d")

Change the -1d according to your needs, run the above search over at least twice the scheduled interval.

koshyk
Super Champion

We had similar siutation and a heartbeat from the other system comes once in every 5 minute. So I'm doing that as a "saved Search" and run every 1 minute and checks data for the previous 6 minutes to see if there is a heartbeat. Else alert.

The logic is:

index=myindex earliest=-6m@s <your_some_more_Search> | stats count as COUNT_HEARTBEAT | where COUNT_HEARTBEAT=0 
0 Karma

MuS
Legend

Hi ashari,

check out the Splunk Deployment Monitor there you can find some nice saved searches related to this topic ... keyword: missing forwarders

hope that helps ...

cheers, MuS

0 Karma

MuS
Legend

sure, although there are a lot of search macros in the saved searches. If you look at the searches you will get an idea how it can be done.
Otherwise use a lookup file which contains all your forwarder hostnames and check if all of them were seen in the past x minutes. Take a look at this http://answers.splunk.com/answers/73268/search-for-hosts-in-a-lookup-but-not-in-splunk

ashari
Explorer

can we do this without splunk deployment monitor app.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...