Can other server types (clustered indexers, heavy forwarders, etc.) fetch config on initial startup (non-search heads)?

[krisreeves]

Search heads have a config option conf_deploy_fetch_url under shclustering in server.conf that causes them to, on startup, fetch the current config bundle from a deployer. Is there any way to make other server types (clustered indexers, heavy forwarders, etc.) do this?

I'm hoping to continue to use the deployer/deployment server/etc. commands to deploy configuration changes, but I would like for new servers with a Splunk instance to bootstrap and configure themselves if possible. Is there any way to do this for indexers? Heavy forwarders? Perhaps other server roles as well?

My fallback is probably something like server-side rendering in React: initially, deploy the config changes to disk in the same location they will be pushed to, and then when settings change, continue to use the deployment server/deployer/cluster master/etc. to push updates.

[rmjharris · Accepted solution]

I'll try to be concise as this could be a long answer. When it comes to bootstrapping a Splunk instance you have three different scenarios: search head cluster member, indexer cluster member and everything else.

Search head cluster member - Here you have to add the [shclustering] stanza to server.conf. This where conf_deploy_fetch_url is set. This is just the url of the Deployer. But you aren't done yet, the search head needs to be added to the cluster. For that you need the "splunk init shcluster-config ..." command. Now the search head has joined the cluster and will pull everything it needs from the captain.

Indexer cluster member - This is the easiest. In server.conf add the [replication_port://] and the [clustering] stanzas. Once it connects to the cluster master it will pull the bundle and restart if necessary. The "splunk apply cluster-bundle" command is used on the master to push out the new bundle to the cluster members.

Everything else - A non-clustered indexer, search head, forwarder or heavy forwarder can use a deployment server. Bootstrapping forwarders is a common practice in Splunk and there is Splunk documentation on it here. In this case you need to configure deploymentclient.conf. The caveat is that the deployment server needs to know what to deploy to that new instance. This is where serverclass.conf (on the deployment server) comes in. There you would have separate stanzas for each class, i.e. [serverClass:MyIndexers]. In this stanza you would whitelist or blacklist the clients so they receive the correct apps. Also, there is the "restart splunkd" item which can be used to for the client to restart. In deploymentclient.conf it is also a configuration item with "reloadDSOnAppInstall". A deployment client is very easy to install. The basic install is just:

[deployment-client]

[target-broker:deploymentServer]
targetUri= deploymentserver.splunk.mycompany.com:8089

Taken from deploymentclient.conf.

Just make sure the deployment server knows what apps to send it.

[krisreeves]

Perfect, this is exactly the info I was hoping for all in one place 🙂 One followup question:

The docs do say that indexers and search heads can use a deployment server, with caveats, but I couldn't work out the caveats. You imply here that it doesn't work for clustered servers, is that true? Or could clustered search heads / indexers also use a deployment server?

Thank you!

[gjanders]

You can use the deployment server to deploy to the search head deployer / cluster master server if you utilize the stateOnClient = noop, from there you can push out the apps to the search heads or indexers.

However you should not connect the search head members or indexer cluster members/peers to a deployment server.

[akira_splunk]

Hey Kris,
NO Deployment server for your Clustered indexers or Search Head Clusters. In your case, you need to use the Cluster Master for the indexers, and a Deployer for your search heads.

A deployment server would only be an option if the indexers were 'distributed' (i.e. when clustering is NOT enabled, but data is spread and load balanced across multiple indexers). And similarly for Search Heads. Only if the SHs are independent. It's not recommended to have the search heads disparate, as user changes done one SH would NOT be replicated to the other search heads.

[krisreeves]

That's what it seemed like, thanks for confirming 🙂

[adonio]

hope i understand your question,
for indexer, if they are clustered they will pull the latest bundle from master when restarting
for forwarders or splunk instances using deployment-server, you can use wild cards in the white-list box, on a particular servercalss. the moment a new instance is up and has the relevant deploymentclient.conf, it will connect to the deployment server, add itself to the server class and as an outcome will "pull" all relevant configuration.
plan carefully

[krisreeves]

I was reading about the deployment server/client thing, and it seems indexers and search heads can use that method too.

I know that you can use 'splunk apply cluster-bundle' to push out a config, but I'm not sure if a new installation of Splunk, configured only to act as an indexer, will put itself in order; I've experienced weirdness along those lines with clustered search heads too, where restarting seems to pull config but not necessarily apply it. It seems like the deployment server is the way to go for automatically deployed hosts like ASGs.

Do you happen to know which configurations cannot be applied with a deployment server? Do I have to set each server's role separately before enabling it as a deployment client or what?