I didn't find "Splunk Add-on for Unix and Linux" on my installation, nor was it installed explicitly by any of the Admin users.
However, there was another plugin present in /opt/splunkforwarder/etc/apps/ by default, called "SplunkUniversalForwarder". Inside the plugin directory, there is a default directory that contains an inputs.conf file. This file contains the monitor statements for $SPLUNK_HOME/var/log folder.
################################
# Make sure these get forwarded
################################
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true
_TCP_ROUTING = *
index = _internal
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true
_TCP_ROUTING = *
index = _internal
I disabled them and Bingo, Splunk stopped flooding the destination with Splunk logs, while sending what I asked Splunk to forward.
Pretty Nice, problem solved. Thanks Pickerin and Lguinn for taking time to discuss and helping me pin point the problem.
However, it is really strange to see that any plugin can change the overall behavior of Splunk as a whole. Wouldn't that be cool, if a changes in the DEFAULT of any plugins, only affect that Plugin and not all applications as a whole ?
Thanks for the help.
-Vipul.
... View more