When managing large numbers of deployment clients, 1000+, what is the optimal check-in interval for those clients?

[sgarvin55]

At times our Splunk Deployment server is unresponsive while other times it's slow to push out changes.

We see Error messages with the following:

DeploymentClient - Unable to send phonehome message to deployment server. Error status is: not_connected"

[sgarvin55 · Accepted solution]

This number can vary depending on the exact numbers of clients and app changes, as
well as how many of the clients are hitting the deployment server at the same time.

A rule of thumb being no more than 2000 clients hitting within a minute – after that –
we have to start increasing the phone home interval. Generally somewhere in the region of
600 - 1200 seconds should be a good place to start. Deployments of 3000 potential clients
should have a phone home interval of 5 - 8 mins.

We do have customers with 15,000 clients who adjust 'phoneHomeIntervalInSecs' and 'handshakeRetryIntervalInSecs' parameters to be successful.

You can also specify a different check-in interval on a per client, or per class basis by
tuning the 'phoneHomeIntervalInSecs' and 'handshakeRetryIntervalInSecs' parameters in
deploymentclient.conf - http://docs.splunk.com/Documentation/Splunk/latest/Admin/Deploymentclientconf

[dstaulcu]

I have many deploymentclients and have set phoneHome and handshakeretry to prescribed intervals. I am still getting non_connected issues on some deploymentClients. What sort of things should I be evaluating on the deploymentServer to troubleshoot?