Splunk Search

What is the correct way to edit conf files and propagate changes in a search head cluster ?

sylvainlectra
Explorer

Let's say I've made an action that triggers configuration replication across the SH Cluster (e.g: created a field extractions through Splunk Web).

Now let's say Splunk Web does not allow me to edit what I've done (e.g. change the owner of the field extraction) and thus oblige me to edit the configuration files manually.

After editing the files on one SH Cluster member (must I do it only on the Captain ?), what should I do to propagate the manual changes across the cluster ?

0 Karma

maciep
Champion

Is that example scenario based on a real one? You should be able to change ownership via REST I think....which will propagate the changes. Manually modifying a conf file will not.

I haven't run into many scenarios in my clusters where i created something that Splunk didn't let me edit, so was curious how common this for you.

0 Karma

koshyk
Super Champion

Depends on how strict you want to make your Splunk system.
(1) if you want to be very strict and end-users NOT to edit anything major, always do via Deployer. This is very good to manage and you can do development and test in similar code-base as PROD. Also the next release will ensure it will backup default etc. so fairly consistent.
(2) if you end-users are doing changes directly via UI (eg creating dashboards, Use-cases etc.) , then their changes will override whatever you do via deployer. This is because when you use deployer, the changes go into "default" directory of the SH members, but the changes via UI will be in "local"

I always consider SHC to be a pain unless you decide on which (1) or (2) option your company looks for.

nnmiller
Contributor

You should not be editing back end configuration files manually on your SHC. You should be performing back end edits on the deployer, then pushing the changes to the SHC.

Please review the following documentation:
Use the deployer to distribute apps and configuration updates
How configuration changes propagate across the search head cluster

There are isolated cases where you may want to change settings in $SPLUNK_HOME/etc/system/local/ config files, but these are rare, and generally only necessary if there is a problem in the SHC.

sylvainlectra
Explorer

So, in a few words, if I do something in Splunk Web but Splunk Web does not allow me to edit what I've done I'm doomed forever ?

0 Karma

mwk1000
Path Finder

I feel your pain, the deployer will throw EVERYTHING in default ( ie. not editable ) . I have had some success at fixing config problems with finding the captain, editing the file, reloading ( NOT reboot or you loose the captain - debug refresh ) the captain, and force syncing all members to the captain. UGLY.

I think the best bet at this stage of search head cluster maturity is to make the rest API your friend and use the config file calls that allow updates to elements, stanzas, or the whole thing. These would be "legal" synced by raft calls to the cluster.

I'm sure it will get better with time, 6.6 just added a fix for ownership that used to kill us when someone left and their ID was no longer available ( yet they owned popular knowledge objects )

brywilk_umich
Path Finder

Instead of changing it on one of the search heads manually, could you just move it up to your deployer in a app and push it to your cluster?

0 Karma

sylvainlectra
Explorer

Not really, I'm not going to ask Splunk users to come to me each time they want to create something on the search heads.

0 Karma

woodcock
Esteemed Legend

It is either/or. Either you allow GUI changes and NEVER EVER use the Deployer to push, OR you use the Deployer to push and NEVER EVER allow GUI changes.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...