Alert (with a chart) when latest occurrences of some of the rows is over a certain time

[curtisk]

New Splunk user here. I've searched and seen a lot of questions and answers that were similar to mine, but none that were close enough to help me figure out what to do in my exact scenario. So, here goes.

I have a number of hosts that have a log file that contains a heartbeat entry that typically occurs every 2 minutes. I created a chart that shows a list of each of the server hostnames along with the time of the latest heartbeat log line and a calculated value of the time since that latest heartbeat was found. This is so you can easily see if it's way over 2 minutes and know you've got some kind of issue with this server.

My challenge now is that I'd like to set up an alert for when the latest time of any server's heartbeat is greater than 30 minutes so a user can get an email with that nice chart embedded, can easily see which server is affected, and how long it's been since the last heartbeat. I just can't figure out how I can do that with the data in chart form... I know the solution is probably easy, it's just eluding me for some reason.

Here's what my search currently looks like, with some index/sourcetypes removed since it's not relevant:

"Sending heartbeat request" earliest=-4h | eval time_elapsed = round((now() - _time)/60,0) | stats latest(_time) AS "Latest Heartbeat", latest(time_elapsed) AS "Minutes since last heartbeat" by host | convert ctime("Latest Heartbeat")

And here's what my output looks like:

I've tried things like added a WHERE clause, but it just doesn't seem to output in the way I need it to. I'm hoping someone's got an easy solution. I'm sure there is one, and my noob-ness is just getting in the way. Thanks!!!

[Richfez · Accepted solution]

Use the search you already have and save it as an alert using a custom search as the trigger.

Click Save As, then Alert
Name it and whatnot
Set an Alert Type of Scheduled
Change the schedule drop-down to Run on Cron schedule
You can set your earliest -4h here if you want, or just leave it in your search
For your Cron Expression put in */15 * * * * to run it every 15 minutes, or */5 * * * * for every 5.
In Trigger Conditions, select Trigger alert when option of type Custom
For the custom search, use search "Minutes since last heartbeat" > 30

You may want to throttle it, maybe not - just depends on your time frame and how often it runs.
Then set whatever alert action you want to use for testing and see what happens in a few minutes.

Did I mention to do a little testing before you shoot some unsuspecting coworker a zillion emails because you fat fingered something? If I didn't, consider it mentioned now. Lessons learned the hard way... 🙂

A link to cron expressions in Splunk, and another on Alert trigger configuration. Other items in the menus on that last link go to examples of alerts, too.

If you'd like to test that and see what it does, you could append that same snippet to the end of your existing search, like so.

... | search "Minutes since last heartbeat" > 30

[Richfez]

Oh, and if you email it, be sure to click the Include option of Inline Table to include the pretty table of results in the email itself.

[curtisk]

BRILLIANT. I didn't realize you could do a secondary search of the results of the table like that! Mind blown. Thanks!!! And yes, I'm testing out sending the email alerts only to myself for now. 😉