Deployment Architecture

Scheduled searches and alerts on Cluster and Search Head

trodenbaugh
Explorer

I'm evaluating moving to a clustered configuration and utilizing the search head. I'm trying to determine how the search head manages scheduled searches and alerts. Specifically where is the savedsearches.conf file located and how do we allow others to create new saved searches and update those saved searches? How does the search head then manage the scheduling of the scheduled searches and alerts?

Regards,
Tom

0 Karma

gbowden_pheaa
Path Finder

Now that I have moved to a search head cluster from a search head pooling (v6.1.1 to v6.2.1), I am getting multiple sent alerts for a single alert. I was able to control this in 6.1.1 by enabling only 1 search head to send e-mail, but would this approach work in a cluster?

I am confused because I have 3 search heads in the cluster, but the cluster sends 2 of each alert, not 1 or 3 as I would expect.

Is there a way to determine which search head actually sends the alerts?

martin_muellar, would you explain why you feel the configuration in a cluster is irrelevant? It was my understanding the SH cluster captain would manage this, but I obviously have a disconnect somewhere.

Also - how should app objects created by users, specifically alerts, be managed if differing configurations are used to control this situation?

Thanks to all in advance.

0 Karma

jeremiahc4
Builder

Kind of a late add, but there's a known problem with multiple search heads sending alerts that was fixed somewhere around 6.2.4 - 6.2.6 release. It fixed the problem with our search heads, but I'm searching for a new problem where our indexers are sending alerts when they shouldn't be.

0 Karma

jnicholsenernoc
Path Finder

gbowden, you can tell which search head sent the alerts by updating the alert_actions.conf file and setting the hostname to be something uniquely identifiable. That's how you can tell.

Just a guess based on past issues like this, are all your clocks NTP'd or sync'd? Sounds like one may be a head.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The savedsearches.conf is stored on the cluster's search head in the same place you store it now on your current search head (which may be a combined search head & indexer instance if you have a single standalone splunk server).

Others add and edit saved searches in a cluster as they do with a standalone server.

Scheduling and alerting works the same way as well, the search head runs a search on a schedule and possibly triggers alert actions. Whether it performs a distributed search or only searches its own index is fairly irrelevant.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...