shc peering and other questions...

[a212830]

Hi,

I'm setting up a pre-prod SHC and have some questions on best practices and such.

My existing indexers are clustered. How should I setup distributed searching? Do it from the deployer? Or from the actual SH instances? If from the deployer, where? The gui, or a props file?

And regarding the deployer and props... if I am updating/creating a .conf file on the deployer, is it still best practice to put these in their own "app", and have the deployer push them out, or should we update the ../etc/system/local entries on the deployer and let it push them out instead?

[maciep · Accepted solution]

My 2 cents...

It might help to take a step back to be sure you understand how search head clustering works at a high-level.

The deployer is not part of the cluster. It is used to push config to the cluster members. With that in mind, the only apps/config on the deployer that get pushed to cluster members are under etc/shcluster/apps (and users). That config gets copied to etc/apps on the search head member. Config changes outside of that directory structure on the deployer won't get pushed to the cluster. So for example, modifying props.conf under etc/system/local on the deployer will only affect the deployer itself. The cluster members won't know anything about those.

Also, it's important to note that apps under shcluster/apps can include default and local settings. When the config is pushed to cluster members, the precedence rules are applied at the deployer level and the resulting config is pushed to the default directory on the cluster members. If changes are then made in splunk web or via rest etc on a search head members, those changes are stored under that app's local directory which gets replicated. That can make troubleshooting a bit tricky, so it's important to keep in mind.

My recommendation would be to store your config various apps, yes. We like to keep each app pretty narrow in its scope to give us more flexibility when modifying/testing changes. For example, we have an app for just the outputs config. Another for the license master. Maybe one for common search head config. Etc. And then of course, you'll have the "actual" apps you'll be pushing to the cluster as well.

Another piece of advice might be to have a sort of staging server. Some apps require further set up after install. And others just various config - maybe disabling searches you don't need, or modifying accelerated data models, etc. It's best to get all of that done on a standalone server. Once it's set up and functioning the way you want, copy it to shcluster/apps on the deployer to push out.

And some links that might be helpful

http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/PropagateSHCconfigurationchanges
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configuresearchheadwithserverconf

[a212830]

Perfect! Thanks for the detailed answer.

[sloshburch]

This is an amazing answer!