How would you deploy 4.2.1 Splunk and Universal Forwarder on a Search Head node that is doing distributed search and also has the *Nix application installed as well? Both the Universal Forwarder and the Splunk deployments use the same daemon name ‘splunkd’. I am deploying UF as a system image along with Splunk for the Search Head. When starting UF I get the message that mgmt port 8089 is already bound. Should I just assign the UF mgmt port to another value? How do I get UF and splunk to co-exist on the SHD? Am I correct in assuming that I will need the UF on the SHD to forward *Nix stats to my Indexer? What recommendations would you make for this configuration?
You won't need the UF to forward to the indexers, that can be done by the SH itself if you configure it from the Manager's forwarding/receiving configuration panel:
You'll also need to create the "os" index on the indexer as those data won't be stoerd on the SH anymore.
Also, override the default system/default/outputs.conf configuation in case you want to ship your SH splunk logs to the indexer as well (index=_intenal is not forwarder by default). But that is another topic...
Running the UF on another port should do the trick I think. Just add the following to ${SPLUNK_HOME}/etc/system/local/web.conf :
[settings]
mgmtHostPort = 127.0.0.1:8088