Why did adding an indexer to our deployment server uninstall all of our custom applications after the required restart?

[ebaileytu]

After a time of constant change to deal with issues I am rebuilding our deployment server using all defaults configurations. I have been adding clients to the deployment server with no issues until I added an indexer and much to my dismay I watched all of our custom application get uninstalled after the required reboot.

I used the set deploy command from the cli to add all the clients in the normal way. Very typical and worked great for 38 hosts.

I restored the custom applications from backup so the impact was not a big deal, but now I am concerned about adding more clients. Anything I can do to make sure this does not happen?

Does the deployment server have a noop option?

Thanks!

[cpetterborg]

It doesn't appear that you are sending anything to the [serverClass:test] machines. That would make those whitelisted servers think that they should not have anything from the deployment server and will remove anything in the etc/apps directory that would be controlled by the deployment server (under the etc/deployment-apps directory on the deployment server).

You should have a list of apps that you want on the indexer, something like:

[serverClass:test]
whitelist.0 = test*
[serverClass:test:app:appnumber1]
[serverClass:test:app:appnumber2]

The two lines that I added are related to the servers listed in the whitelist because they have the same serverClass:test at the beginning of the definition. when the deployment server is contacted by a whitelisted server, the apps (appnumber1 and appnumber2) will be sent to them. If they change at some point, the deployment server with then send the updated files. If you were to remove one of those apps from the configuration, the server would be told to remove that app by the deployment server.

You can do all the deployment of all the apps at once, it should not make a difference. You just have to get the configuration right.

If you are using clustered indexers, don't use the deployment server for the deployment of the apps to the cluster master, just make your modifications on the cluster master. Then when you apply the cluster-bundle it will distribute to the indexers and then do a nice rolling restart.

[cpetterborg]

Can you supply the configuration (serverclass.conf maybe) that you used when you added the indexer? If you have before and after, that would be even better.

[ebaileytu]

sure

before

[global]
repositoryLocation = $SPLUNK_HOME/etc/deployment-apps
targetRepositoryLocation = $SPLUNK_HOME/etc/apps
tmpFolder = $SPLUNK_HOME/var/run/tmp

stateOnClient = enabled

restartSplunkWeb = False
restartSplunkd = False

continueMatching = true

after adding first client

[global]
repositoryLocation = $SPLUNK_HOME/etc/deployment-apps
targetRepositoryLocation = $SPLUNK_HOME/etc/apps
tmpFolder = $SPLUNK_HOME/var/run/tmp

stateOnClient = enabled

restartSplunkWeb = False
restartSplunkd = False

continueMatching = true

[serverClass:test]
whitelist.0 = test*

[ebaileytu]

Is it advisable to incrementally move apps and configs to a deployment server or does it have to be done all at once? I am going to be cautious moving forward if I get this sort of behavior since it can cause a major disruption.