Issue with light forwarder on cloned hosts

[emiller42]

We recently added several hosts that would be forwarding data to our indexers. Since all the hosts were going to be cloned off of a template, we thought that to save time, we would simply install the forwarder on the template, give it the proper deploymentclient.conf. That way, when we cloned it, the clones would simply poll the deployment server and be up and running without any manual intervention.

This seemed to work, as the clones did pull the appropriate application bundles from the deployment server. But we aren't seeing forwarded data on the indexer. Checking splunkd.log on the forwarders shows errors where the indexer would refuse the connection, but we can't see any reason why. All configs are correct, and there are no connectivity blocks between the forwarder and the indexer.

Any ideas of what to check next?

[emiller42 · Accepted solution]

Answering this one myself in case it helps someone else.

I just discovered that the installer for the light forwarder creates an inputs.conf in etc/system/local that specifies the host name. Since we ran the installer on the template, this .conf file had the host name of the template. Then when we cloned the template, all of the clones had that conf file in place, setting their host name to be identical to the template.

Since the connection refusals from the indexer were due to it getting what appeared to be 30+ simultaneous connections from the same host.

Modifying the conf file to have the appropriate host name and restarting the forwarder corrected the issue.

[emiller42]

Good info, thanks!

[kristian_kolb]

you might also ensure that GUID is unique for all the cloned forwarders, since that is what e.g. the DeploymentMonitor uses to separate the forwarders from each other. I learned this the hard way 🙂

http://splunk-base.splunk.com/answers/32368/duplicate-guids-for-cloned-forwarders-how-to-correct

Also there is also a section of the docs relating to cloning; http://docs.splunk.com/Documentation/Splunk/latest/Admin/PutSplunkontosystemimages and the pages that follow it.

/k