Running into a strange scenario with host field resolution in combination with Splunk App for Stream. All the forwarders are pushed to end-points using chef and they currently copy over the inputs.conf from /etc/system/default into /etc/system/local adding a couple of monitor stanzas to the one in local to monitor additional files. This carries over the default stanza with host=$decideOnStartup into the inputs.conf under local as well
[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup
The monitor inputs however, do get the resolved (correct) host information, as per below. Once the Splunk App for Stream is pushed to the forwarders, the stream data though picks up the host as $decideOnStartup unless either the host is explicitly set in the inputs.conf under /etc/system/local or the forwarder is cleaned up, and no inputs.conf from /etc/system/default is copied over to /etc/system/local which then effectively means the forwarder creates one at startup with the resolved hostname under which then keeps things humming along.
Question is, how are the monitor inputs able to resolve the host=$decideOnStartup to the correct host name, even when placed under /etc/system/local while the Stream input is not? Anyone seen this before and have suggestions on how to get past it, without having to either (a) manually set the host field or (b) redeploy the forwarders without a copy of the inputs.conf from /etc/system/default
... View more