How do you make sure Search Head Cluster objects are synced in all nodes?

[ben_leung]

If someone were to modify the savedsearches.conf via the CLI in one of the nodes of a search head cluster, what would be the way to make sure that node is in sync again? Is there a command to run to check and rsync? What would be the process to make sure things are synchronized?

[Lucas_K]

Do a splunk search 🙂

In your example you are looking for a savedsearch modification but you can do it for any object.

index=_internal source=*conf.log | xmlkv  | search "data.optype_desc"=WRITE_STANZA data.asset_uri{}=*savedsearches* | table _time host data.from_repo data.asset_uri{} data.task | stats values(host) AS host count by data.asset_uri{}, data.task | addcoltotals labelfield=Total count

Total count should be the number of members in your search head cluster.

[rsigle]

splunk resync shcluster-replicated-config does a destructive resync, but in a way it does what you are looking for. If you edit savedsearches.conf on the captain and run splunk resync shcluster-replicated-config, the change will be replicated, but again, this is a destructive resync, so if a user makes a change via the UI on one of the nodes that is not currently the captain, the changes could be lost.

We've used this method to migrate apps/users from standalone search heads to the search head cluster during maintenance periods.

[ben_leung]

Is this the command I may be looking for?

splunk resync shcluster-replicated-config

[pradeepkumarg]

You should not make changes on the member directly using CLI. Either you should do it via Web on the member or through the deployer

http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/HowconfigurationworksinSHC

[Steve_G_]

That's not true. The cluster should be replicating changes made via the CLI. See: http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/HowconfrepoworksinSHC#Configuration_met...

[ben_leung]

I need a command for sanity check. Who knows what could go wrong via the filesystem or if a bad user decides to modify something, could be accidental.

[pradeepkumarg]

Right, I meant specific to knowledge objects. The same documentation has this

The cluster does not replicate any configuration changes that you make manually, such as direct edits to configuration files.

For example, if a user creates a saved search in Splunk Web on a cluster member, the cluster replicates that saved search to all cluster members. However, if you, as the administrator, add a saved search by directly editing the savedsearches.conf file on one cluster member, the cluster does not replicate that saved search to the other cluster members. You must use the deployer to push that saved search to all cluster members.

[ben_leung]

Is there a command on the captain where I can manually run to check indefinitely that the other nodes are in sync?