Alerting

Triggered Alerts are Occuring twice on every Event that happens once

sbeamro
Explorer

I have configured an Alert that is running in real time.
with the value of host="10.56.183.0" "%LINEPROTO-5-UPDOWN"
since 10.56.183.0 is a switch and I'd like to recieve an email when interface goes up or down.

When the event occurs once (I can see in the search that it ocurrs once) I'm getting 2 emails.
and when looking at the Alert window - I can see that it counted 2.

any idea why ?

Tags (1)
0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

That being said, it might be generally useful (regardless of version) to consider throttling the alert. Real-time alerts with per-result triggering can sometimes fire more often than you need. For example, our docs have this note about this alert type:

"You can also use transforming commands to return results based on processing the retrieved events. A per-result alert triggers in both cases, when the search returns an event or when a transforming command returns results."

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/Defineper-resultalerts

Although I don't see a transforming command in your query example, there might be some similar behind-the-scenes processing of the initial retrieved events that is causing the extra triggering.

Here is some documentation about how to throttle an alert to reduce the frequency of alert triggering:

http://docs.splunk.com/Documentation/Splunk/6.3.1511/Alert/ThrottleAlerts

Hope this helps!

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Hi there,
I think at this point, if there is still unexpected behavior with the alert, the best option for getting more troubleshooting help would be to contact Support. They can walk through your data and alerting configuration more specifically to try and figure out what's going on. Here is some contact information:

http://www.splunk.com/en_us/about-us/contact.html#customer-support

Hope this helps! Good luck!

0 Karma

dschnabel
Explorer

Thank you for looking into this and providing some ideas. I'll get in touch with customer support.

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

You're welcome!

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Also--to dig a bit more into how your query is returning events and to see if there is a way to modify it to establish more efficient triggering, could you post some info from the results that the query returns? For one event, could you post examples of what's in the following fields?

_raw
clientID
class

0 Karma

dschnabel
Explorer

I'll give you the _raw data which contains all the fields of an event:
+++ 2016-01-12 17:57:28 +++
ClientId=my-clientid
HostId=my-hostid
Hostname=my-hostname
Platform=Linux (Ubuntu 12.04.5 LTS)
Versions=app1:1.3.3,app2:1.2.0
Mode=Installation
Class=PROCESS_NOT_RUNNING
Msg=One or more processes are not running

(I've replaced some field values since they contain sensitive data)

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Ok, sounds like throttling fixed the double triggering. Perhaps there's a way to rewrite the query to deal with the other aspect of the problem.

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Going back to your query:

index=tv- ClientId NOT (some-id-1 OR some-id-2 OR some-id-3) ClientId="" Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

If I'm interpreting the query correctly, then the events you want to alert on should follow these parameters
--ClientId== anything other than "some-id-1", "some-id-2", or "some-id-3"

--Class== not "success_first_attempt" or "Server did not accept key"

--Mode== only installation

If the above is correct, what happens if you remove this from your query: Client = "*"?

0 Karma

dschnabel
Explorer

--ClientId== anything other than "some-id-1", "some-id-2", or "some-id-3"
That's almost correct, see below.

--Class== not "success_first_attempt" or "Server did not accept key"
Class== not "success_first_attempt" is correct, the phrase "Server did not accept key" would appear in the 'Msg' key. So properly this part would be:
--Class== not "success_first_attempt" and --Msg== not contain "Server did not accept key"
but it also works if 'Msg' is omitted.

--Mode== only installation
That's correct

what happens if you remove this from your query: Client = "*"?
ClientId can be empty (ClientId=). In that case we don't want to trigger the alert. That's why I have the expression ClientId="*" in there.

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Give this alternative query a try:

index=tv- (ClientId=* AND ClientId NOT (some-id-1 OR some-id-2 OR some-id-3)) Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

0 Karma

dschnabel
Explorer

Ok, and what should I set for "Per result throttling fields"?

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

I'm running a couple of questions about your query by my colleagues who work on the search language. You could try the query without throttling, to see if it yields the results you want. I'll report back if there's further advice we can offer! Stay tuned.

0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Ok, my colleagues got back with a slightly different query suggestion. Try it without throttling to see if it yields the results you want.

index=tv- ClientId !="" ClientId NOT (some-id-1 OR some-id-2 OR some-id-3) Class NOT SUCCESS_FIRST_ATTEMPT NOT "Server did not accept key" Mode=Installation

The difference is in using ClientId !="" vs. ClientId=. According to my colleagues, if you use ClientId=, you will get events that have this field, even if the value for that field is an empty string. Try !="" to avoid the events with empty ClientId values.

0 Karma

dschnabel
Explorer

Ok, I've updated the query and will wait a few hours to see how it works. Will let you know.

(btw there should be only one closing bracket instead of two in your suggestion)

0 Karma

dschnabel
Explorer

@frobinson unfortunately I'm seeing two alerts again now:

  • I see two alerts if ClientId NOT (some-id-1 OR some-id-2 OR some-id-3)
  • I see one alert if ClientId = some-id-3 (<-- no alert should be triggered at all in this case)
0 Karma

frobinson_splun
Splunk Employee
Splunk Employee

Got it. Please also feel free to accept answers if they've helped with your question.

0 Karma

dschnabel
Explorer

Ok, I've enabled throttling and will let you know soon if that works.

frobinson_splun
Splunk Employee
Splunk Employee

Yes, please do!

0 Karma

dschnabel
Explorer

The double alerts are gone. Instead I'm now seeing event alerts that shouldn't trigger because 'ClientId=some-id-3' and my query (see above) should filter those out. But that's a different problem.

0 Karma

dschnabel
Explorer

I'm seeing the same behavior with an alert that I configured. The alert is triggered twice but the event only happens once. To be safe I deleted and re-created the alert but the problem didn't go away. Is this an issue with Splunk alerts?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...