Deployment Architecture

Custom whitelist for Search Head Cluster Replication

dilipbailwal
Path Finder

i have been trying to create a custom white list for Search Head Cluster Replication under one of the cluster member at "splunk_home/etc/system/local/server.conf ", but it seems not working.

Here is the content of server.conf

Custom changes replicated

conf_replication_include.web = true

where, i am trying to add web.conf to be replicated across. itself.. Even this file is located at "/splunk_home/etc/system/local/web.conf"

Also i did rolling restart after its default 5 second window did not replicated..
./splunk rolling-restart shcluster-members

hence still no sign of file getting replicated..

0 Karma

steven_swor
Path Finder

I've deployed several apps over the last couple weeks in a SHC environment, with extra configs that needed to be replicated. My experience so far has been that:

  1. splunk rolling-restart shcluster-members does not cause the new replication configs to be loaded from server.conf
  2. Once the extra replication configs have been properly loaded by all the cluster members, the replication is only triggered when you make changes via the web UI (e.g. if there already was a $SPLUNK_HOME/etc/system/local/web.conf, it won't be replicated until you change it again via the web UI)
  3. If the app requires configuring a password as part of the setup process, the passwords may not be replicated across the search head cluster (even if the rest of the config file is replicated), so you have to log into each search head in the cluster and set the password explicitly. I saw this when setting up SA-ldapsearch, Splunk App for VMWare, and Splunk App for NetApp Data ONTAP.

The most reliable way I've found of ensuring the new conf_replication_include... settings are loaded properly has been to undeploy an app from the search head cluster and then redeploy it. On the deployer:

  1. Move an app from $SPLUNK_HOME/etc/shcluster/apps to a temporary location
  2. Run splunk apply shcluster-bundle ... to roll out the search head cluster apps
  3. Move the app back into $SPLUNK_HOME/etc/shcluster/apps
  4. run splunk apply shcluster-bundle ... again to redeploy the bundle

Note that this will trigger two rolling restarts of the search head cluster (one when you undeploy the app and one when you re-deploy it). After this is done, the replication settings should take effect the next time you make a change through the web interface.

As a best practice, I'd recommend creating a "poison pill" app on the deployer. The app itself has all the required folder structure and contents to be treated as a valid Splunk app, but it does nothing (e.g. no inputs, dashboards, saved searches, et cetera). You can undeploy and re-deploy it at-will, to trigger the restarts properly, without losing important configs from "real" apps.

0 Karma

dilipbailwal
Path Finder

All rite Steven, i am gonna give that a shot by creating a poison pill

0 Karma

dilipbailwal
Path Finder

it was case typo in question.. not the case with my config files on servers..

jtrucks
Splunk Employee
Splunk Employee

You should mark this answer as "Accepted" so people know what the answer was, as well as to get credit for answering your own question here. 🙂

--
Jesse Trucks
Minister of Magic
0 Karma

dilipbailwal
Path Finder

Its not the answer jtrucks.. and what i am trying to say here is, it was a typo on my question where server.conf was written in upper case.. which i have fixed and the main issue still exist..

0 Karma

jtrucks
Splunk Employee
Splunk Employee

I suspect these file names are case sensitive in the configuration, so rename your file server.conf instead of SERVER.CONF and try again.

--
Jesse Trucks
Minister of Magic
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...