Why isn't splunkd.log getting forwarded (in 4.0.x, fixed in 4.1) ?

[oreoshake]

UPDATE: This appears to be a bug specifically related to 4.0.10. The following is a work around in system/local/inputs.conf

[monitor:///home/ops/splunk/var/log/splunk]
disabled=true

---

It looks like the entry in system/default is clobbering the more granular entries in SplunkLightForwarder/default. The output of splunk list monitor seems to be broken as well

I'd like to index the splunkd.log for remote troubleshooting but I can't get my light forwarders to forward the log. Here's the btool output (with unrelated stuff removed):

[root@neil bin]# ./splunk cmd btool --debug inputs list
system     [default]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = default
system     [monitor:///home/ops/splunk/var/log/splunk]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = _internal
SplunkLigh [monitor:///home/ops/splunk/var/log/splunk/splunkd.log]
SplunkLigh _TCP_ROUTING = *
system     _rcvbuf = 1572864
system     host = myforwarder
maint      index = _internal
maint      sourcetype = splunkd
atti-linux [monitor:///var/log]
system     _rcvbuf = 1572864
atti-linux _whitelist = (auth$|cron$|kern$|lpr$|maillog$|user$|local$)
system     host = myforwarder
system     index = default
system     [splunktcp]
system     _rcvbuf = 1572864
system     host = myforwarder
system     index = default
system     route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue

I tried changing the index, no dice. From what I can tell, the /var/log/splunk entry clobbers the more specific one which says to include the entire var/log/splunk directory. When I run "splunk list monitor" it prints out the entire var/log/splunk dir except splunkd.log!

[root@neil-search etc]# ../bin/splunk list monitor
Monitored Directories:
    $SPLUNK_HOME/var/log/splunk
        /home/ops/splunk/var/log/splunk/audit.log
        /home/ops/splunk/var/log/splunk/btool.log
        /home/ops/splunk/var/log/splunk/metrics.log.1
        /home/ops/splunk/var/log/splunk/metrics.log.2
        /home/ops/splunk/var/log/splunk/metrics.log.3
        /home/ops/splunk/var/log/splunk/metrics.log.4
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-23.00-14-09
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-23.17-02-12
        /home/ops/splunk/var/log/splunk/migration.log.2010-03-26.18-58-16
        /home/ops/splunk/var/log/splunk/searchhistory.log
        /home/ops/splunk/var/log/splunk/splunkd_access.log
        /home/ops/splunk/var/log/splunk/splunkd_stderr.log
        /home/ops/splunk/var/log/splunk/splunkd_stdout.log
        /home/ops/splunk/var/log/splunk/splunklogger.log

[the_wolverine]

Oreoshake, your reasoning is correct. The reason the SplunkLightForwarder/defaults are not working is because they are overridden by system/defaults. Thus, the desired configuration needs to be moved to a local/inputs.conf.

[the_wolverine]

As mentioned, this could be a bug. It would be useful if you can post the version you are encountering this behavior in so we can pinpoint whether you are encountering a bug.

[the_wolverine · Accepted solution]

For clarification, this is applies to Lightweight Forwarder. The default expected behavior (LWF forwards its internal logs as default) is not working in version 4.0.x. The workaround is to create a monitor input for the Splunk logs directory and whitelist specific files in the $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk]
_TCP_ROUTING = *
_whitelist = (splunkd|metrics|license_audit)\.log$

Note: This is working without additional configuration in version 4.1.