Alerting

alerts triggering continuously

CREVITCH
Path Finder

I am saving the following alerts:

"user=* | search failed | dedup _raw"

real time 60 second window. It triggers only when new event with “failed” is detected.

"user=* | search failed | dedup _raw | stats count by user"

real time 60 second window. It alerts continuously

how do I get the second alert to trigger only when a new event is detected?

Tags (2)
0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Hi @CREVITCH,
I think the problem is the transforming command at the end of the second search. This is causing the alert to fire when events are found and when the stats count command returns results.

Our docs state that this can happen with per-result alerts and transforming commands:
"You can create a search to retrieve events from an index. You can also use transforming commands to return results based on processing the retrieved events. A per-result alert triggers in both cases, when the search returns an event or when a transforming command returns results."

See:
http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/Defineper-resultalerts

To change the triggering behavior, you can consider adding a trigger condition for the count of events, or perhaps throttling:

"If you have scheduled searches that run frequently and you do not want to be notified for each run, set the throttling controls to suppress the alert to a longer time window.

For real-time searches, if you configure an alert so that it fires once for a trigger condition, you do not need to configure throttling. If the alert fires more than once for the trigger condition, consider throttling to suppress results.

When you configure throttling for a real-time search, start with a throttling period that matches the length of the base search's time window. Expand the throttling period if necessary. This prevents multiple notifications for a given event."

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/ThrottleAlerts

Hope this helps!

0 Karma

CREVITCH
Path Finder

can you show me an example of how to do this?

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...