Alerting

How to create an alert on a calculated field

pjbuchan596
Explorer

I am attempting to create an alert based on a field calculated from all of the events of the search, specifically when it is less than some value. However, the alert solutions I've found thus far look to give alerts based on the count of events returned by the search, or filtering on a field of the events to determine the count of events that meet the filter. The calculated field I am interested in creating the alert on is not passed in any of the events. My search string for the alert is as follows (the search itself I've left out as I've already narrowed down to the events I am interested in):

search ...filter...
| eval device_duration = 0
| eval sensor_duration = 0
| convert num(SM_C.value.data.elapsedTime) as device_duration num(SM_C.value.data.time_active) as sensor_duration
| stats sum(device_duration) as total_device_duration sum(sensor_duration) as total_sensor_duration
| eval ratio = if((total_device_duration > total_sensor_duration), round((total_sensor_duration / total_device_duration), 2), round((total_device_duration / total_sensor_duration), 2))
| eval ratio = 100 * ratio

The search is comparing events from 2 devices and getting the difference ratio between the sum of durations reported by each device's events.

I'm looking to send an alert if the ratio falls below some threshold, say 70% (correlation drops below 70%). In the alert itself for the Trigger Conditions I have it set to a custom trigger "eval ratio < 70" but this does not cause the email. Please let me know what I'm doing wrong, or if there is a way to modify the search string to work on a different trigger condition. Thank you for any help!

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi pjbuchan596,
If I correctly understood, add at the end of your search the condition you want (e.g. | where ratio<70 ) so if you find events,alert is triggered, otherwise there's no alert.

search ...filter...
| eval device_duration = 0
| eval sensor_duration = 0
| convert num(SM_C.value.data.elapsedTime) as device_duration num(SM_C.value.data.time_active) as sensor_duration
| stats sum(device_duration) as total_device_duration sum(sensor_duration) as total_sensor_duration
| eval ratio = if((total_device_duration > total_sensor_duration), round((total_sensor_duration / total_device_duration), 2), round((total_device_duration / total_sensor_duration), 2))
| eval ratio = 100 * ratio
| where ratio<70

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi pjbuchan596,
If I correctly understood, add at the end of your search the condition you want (e.g. | where ratio<70 ) so if you find events,alert is triggered, otherwise there's no alert.

search ...filter...
| eval device_duration = 0
| eval sensor_duration = 0
| convert num(SM_C.value.data.elapsedTime) as device_duration num(SM_C.value.data.time_active) as sensor_duration
| stats sum(device_duration) as total_device_duration sum(sensor_duration) as total_sensor_duration
| eval ratio = if((total_device_duration > total_sensor_duration), round((total_sensor_duration / total_device_duration), 2), round((total_device_duration / total_sensor_duration), 2))
| eval ratio = 100 * ratio
| where ratio<70

Bye.
Giuseppe

0 Karma

pjbuchan596
Explorer

When I add "where ratio < x" to the end it stops reporting the statistics when the ratio is less than x, but does not affect the number of events. I suspect this is because ratio is not a field value passed in with the events, and is calculated with the sum of the events. It doesn't look to filter the events themselves.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Yes, you can use ratio to filter results: in a dashboard you have all the values with ratio (without the where condition), instead in an alert you have only the values down the threshold with the where condition.
Bye.
Giuseppe

0 Karma

pjbuchan596
Explorer

I have the alert set so that one of the trigger actions is to add the alert to the triggered alerts list. However, when I run the alert, both with and without "| where ratio<70" appended to the end of the search string, the alert does not trigger. Though from the statistics given when the alert runs the ratio is set to 58, so I would think it would trigger the alert. Any thoughts on what the issue might be?

The Trigger conditions are: Trigger alert when number of results is greater than 0. Set to trigger once, without throttle selected.

Note: Realized after doing some additional investigation of above issue that the alert will not generate an action on a manual run, but only a scheduled run. I was testing the alert by manually choosing run, as opposed to setting a scheduled run.

Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...