Alerting

How to prevent duplicate alerts with multiple search heads?

koshyk
Super Champion

We got a scenario whereby there are multiple search heads. (Say 2x of them). The main reason being load balancing (both active), providing redudancy etc. Hence logging from end-user seems fine.

The challenge with us is, alerting configuration. Splunk searches bulk of data every 2 minutes and sends alerts to Tivoli. Inorder to bring consistency, the same app package is deployed into all Search Heads. Thus the alerting functionality is active at all SHeads. But as a result of this configuration , the alerts are duplicated. (i.e. same alerts pushed from all SHeads)

Is there any way/method to prevent this? We can't disable schedule searches as we don't know in case of a failure no one notices it.

Tags (3)
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Search Head Pooling should do what you need, plus no need to manually deploy the apps to both servers.
For a scheduled search, that includes alerts, exactly one search head in the pool will run an invocation along with any actions executed on the results.

http://docs.splunk.com/Documentation/Splunk/6.1.2/DistSearch/Configuresearchheadpooling

View solution in original post

koshyk
Super Champion

@somesoni2 , but that means the single job server is a single point of failure ?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Search Head Pooling should do what you need, plus no need to manually deploy the apps to both servers.
For a scheduled search, that includes alerts, exactly one search head in the pool will run an invocation along with any actions executed on the results.

http://docs.splunk.com/Documentation/Splunk/6.1.2/DistSearch/Configuresearchheadpooling

koshyk
Super Champion

@martin_mueller thanks again

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, at least that's what supposed to happen. Not sure how Lucas' reported bug may interfere though.

koshyk
Super Champion

@martin_mueller, thanks for that. "For a scheduled search, that includes alerts, exactly one search head in the pool will run an invocation along with any actions executed on the results". Will Splunk automatically switch to next available SH if one of the Head is down for maintenance?

0 Karma

Lucas_K
Motivator

I'm finding that certain real time alerts will run independently on each search head pool. When an email is sent they will be send from each host in the pool.

http://answers.splunk.com/answers/86058/mutiple-alerts-from-scheduled-real-time-search-in-search-hea...

I have a feeling this is a bug in the search head pooling scheduler (logging a ticket). Disabling scheduling on all search heads except for a specific one feels, well, very resource inefficient.

somesoni2
SplunkTrust
SplunkTrust

We also have 8 search heads working for user load balancing. For Splunk object which requires only one instance to be run like summary index searches or alert, we have configured separate single job server, so that we don't run into the very same situation. You might go for something like that instead of putting it on search heads.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...