How to trigger an alert if status event is not indexed for 5 minutes?

[Skins]

I've found a post here - but I'm a bit confused on how to implement this or if there is another method ?

https://answers.splunk.com/answers/475724/how-to-create-a-search-that-will-trigger-an-alert-1.html

I'm monitoring a status event in a log file and want to know if that status stops being indexed.

gratzi

[gcusello]

Hi Skins,
if you have only to know when events flow is stopped you can run a simple search (like index=my_index sourcetype=my_sourcetype earliest=-300s latest=now) and save it as an alert scheduling every five minutes (cron */5 * * * *).

In instead you have to monitor when you don't receive events from an host in a list you have to create a lookup with all your monitored hosts (e.g. perimeter.csv) and run something like this:

| metasearch index=_internal earliest=-300s latest=now
| eval host=upper(host) 
| stats count by host
| append [ | inputlookup perimeter.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0

and then save it as an alert scheduling every five minutes (cron */5 * * * *)
Bye.
Giuseppe

[Skins]

I tried this method - but it doesn't seem to work.

created a lookupfile from a search and it looks like:

host
"127.0.0.1"
"www.destinations.com"

created the search :

| metasearch index=_internal earliest=-1m latest=now
| eval host=upper(host)
| stats count by host
| append [ | inputlookup hosts_list.csv | eval host=upper(host) , count=0 | fields host count ]
| stats sum(count) AS Total by host
| where Total=0

get the following as a result - even though the hosts are generating events every few seconds

host Total
127.0.0.1 0
WWW.DESTINATIONS.COM 0

if i then add a nonsense host to the lookup file ..

host
"127.0.0.1"
"www.destinations.com"
"TREACLE"

i then get the following output even though theres never been an event for the host TREACLE

host Total
127.0.0.1 0
TREACLE 0
WWW.DESTINATIONS.COM 0

[gcusello]

Hi Skins,
are you sure that hosts in your lookup are in _internal?
"127.0.0.1" and "www.destinations.com" don't seem to be hostnames, you can verify this with a simple search index=_internal host="127.0.0.1" OR host="www.destinations.com".
From the result of your search they aren't known hosts.
If you don't put | where Total=0, you have all the events from your hosts.
Verify hostnames and put in your lookup an hostname that you're sure to find in _internal and see if you find events.
Bye.
Giuseppe

[richgalloway]

Create a scheduled search for the event that needs to be there. Have the alert trigger if the search returns zero results.

[skoelpin]

This is the way to go. Click Save As --> Alert and choose equal to No Results. This will say, if no results are returned, then fire the alert

[niketn]

@Skins, you will have to add more details of the event you are trying to monitor. Field name and sample data.

Following is an example of heartbeat query in Splunk Documentation:
https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Addinfo#2._Determine_which_heart...