Alerting

Alert condition with dynamic variables

gallantalex
Path Finder

Hi, I am having trouble in create a condition for an alert that I would like. I have just started using Splunk and I do not know all the fancy search conditions.

So I have indexed results for all our projects that contain information like the number of unit tests failed. Sample events being :

| Project: A | FailedTests: 0| Date: 12062010 | ...
| Project: B | FailedTests: 3| Date: 11042010 | ...

I would like an alert whenever the number of tests failed for a certain project is greater then that of the last value for the number of tests failed. Is there any custom condition that I could use to do this? Thanks in advance.

Tags (1)
0 Karma
1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

I'll assume that you've already extracted the relevant fields and have good time extraction in place. You can achieve this by either using streamstats to compute the differences between subsequent runs for each project or by dedup and stats to find the most recent two runs for each project. I'll provide the dedup solution since it seems a bit simpler to me.

... | dedup 2 Project | stats first(FailedTests) as current_failed last(FailedTests) as previous_failed by Project | where current_failed > previous_failed

Provided that the search is run over a long enough time range, it will find all projects where the most recent number of failures is more than the previous recorded number of failures. You can then configure your alert to trigger when the number of events is greater than zero.

The biggest problem here is that the alert will keep on firing until we have the same or fewer failures for all projects. Perhaps that's desirable. If not, you could use a lookup table to store the previous number of failures per Project. Then you'll have exactly one line in an alert per increase. A search like this will work:

... | stats first(FailedTests) as current_failed by Project
    | lookup failure_count.csv Project OUTPUT current_failed as past_failed
    | outputlookup failure_count.csv
    | where current_failed > past_failed

Here we compute the most recent failure count by project, then we look up the previous failure count (called current_failed in the lookup), then we save our revised table and finally filter out those projects that have a past failure and a higher FailedTests than before.

View solution in original post

Stephen_Sorkin
Splunk Employee
Splunk Employee

I'll assume that you've already extracted the relevant fields and have good time extraction in place. You can achieve this by either using streamstats to compute the differences between subsequent runs for each project or by dedup and stats to find the most recent two runs for each project. I'll provide the dedup solution since it seems a bit simpler to me.

... | dedup 2 Project | stats first(FailedTests) as current_failed last(FailedTests) as previous_failed by Project | where current_failed > previous_failed

Provided that the search is run over a long enough time range, it will find all projects where the most recent number of failures is more than the previous recorded number of failures. You can then configure your alert to trigger when the number of events is greater than zero.

The biggest problem here is that the alert will keep on firing until we have the same or fewer failures for all projects. Perhaps that's desirable. If not, you could use a lookup table to store the previous number of failures per Project. Then you'll have exactly one line in an alert per increase. A search like this will work:

... | stats first(FailedTests) as current_failed by Project
    | lookup failure_count.csv Project OUTPUT current_failed as past_failed
    | outputlookup failure_count.csv
    | where current_failed > past_failed

Here we compute the most recent failure count by project, then we look up the previous failure count (called current_failed in the lookup), then we save our revised table and finally filter out those projects that have a past failure and a higher FailedTests than before.

gallantalex
Path Finder

Our build server has been down for a while so I haven't had the time to fully test this until now. And it worked perfectly, exactly what we needed, thank you so much.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...