Alerting

How to create an alert if no data is generated from a host?

ashari
Explorer

I want to run a search in splunk to find out that all the devices attached to the splunk server are generating logs. If I dont hear from a device I should receive a alert.

thanks

Tags (1)

martin_mueller
SplunkTrust
SplunkTrust

Here's a quick way to check for hosts not having sent data in a while:

| tstats latest(_time) as latest where index=* by host | where latest < relative_time(now(), "-1d")

Change the -1d according to your needs, run the above search over at least twice the scheduled interval.

koshyk
Super Champion

We had similar siutation and a heartbeat from the other system comes once in every 5 minute. So I'm doing that as a "saved Search" and run every 1 minute and checks data for the previous 6 minutes to see if there is a heartbeat. Else alert.

The logic is:

index=myindex earliest=-6m@s <your_some_more_Search> | stats count as COUNT_HEARTBEAT | where COUNT_HEARTBEAT=0 
0 Karma

MuS
Legend

Hi ashari,

check out the Splunk Deployment Monitor there you can find some nice saved searches related to this topic ... keyword: missing forwarders

hope that helps ...

cheers, MuS

0 Karma

MuS
Legend

sure, although there are a lot of search macros in the saved searches. If you look at the searches you will get an idea how it can be done.
Otherwise use a lookup file which contains all your forwarder hostnames and check if all of them were seen in the past x minutes. Take a look at this http://answers.splunk.com/answers/73268/search-for-hosts-in-a-lookup-but-not-in-splunk

ashari
Explorer

can we do this without splunk deployment monitor app.

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...