How to set up an alert to trigger if certain indexes do not receive any events within a certain time range?

[kestasm]

Hello,

We are monitoring a few indexes which are very important for us, but sometimes services get stuck and events are not sent to the indexes. Is it possible to create an alert which would send us a notification when there is no event condition met in an hour or any similar timeline?

I tried to using this, but it seems it doesn't work:

|metadata type=index |search index=A OR index=B OR index=C |eval age=now() - lastTime | search age>3600 

[aaronkorn · Accepted solution]

Try this and adjust your thresholds accordingly then create a search that alerts when it matches your alert criteria:

| tstats latest(_time) AS lastTime WHERE index=_internal OR index=_audit BY index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" max(lastTime) AS "Latest Event" by range index age_min | convert ctime(*Time) ctime("Latest Event")  | rename  age_min AS "Communicated Minutes Ago"

[aaronkorn]

Assuming you're on Splunk 6.x, something like the following should do the trick. Also, do you care about the index as a whole or the different sourcetypes within it?

| tstats latest(_time) AS lastTime WHERE index=A OR index=B OR index=C  BY host sourcetype index | eval current=now() | eval age_min=round((current-lastTime)/60,2) | rangemap field=age_min default=severe low=0-30 elevated=30-60 | stats max(current) AS "Current Time" values(sourcetype) AS Sourcetypes list(lastTime) AS "Latest Event" list(age_min) AS age_min list(range) AS Status by host | convert ctime(*Time) ctime("Latest Event")  | eventstats avg(age_min) AS avg_age by host | eval avg_age=round(avg_age,2) | sort - avg_age | rename  age_min AS "Communicated Minutes Ago" avg_age AS "Avg Communicated Minutes Ago"

[Pawlub1]

I like the search, but what are we using as the trigger.

Typically, something like 'search count < 1' for a zero trigger

I tried 'Communicated Minutes Ago < 1' ,but that is not triggering the alert. 

Cheers, Paul

[kestasm]

Just the index as a whole.