I'm having an problem where the universal forwarder isn't reading any log files except for syslog and messages. I've been looking at this issue for a while and I don't know where to look now.
When I set up the deployment server I organized the input files organized into a global file, web file, and server specific. Here's what they look like:
Global-inputs.conf
[monitor:///var/log/syslog*]
ignoreOlderThan=2d
[monitor:///var/log/messages*]
ignoreOlderThan=2d
[monitor:///var/log/custom/startup/*]
sourcetype=startuplogs
ignoreOlderThan=20d
[monitor:///var/log/custom/backup/*]
sourcetype=backuplogs
ignoreOlderThan=20d
web-inputs.conf
[monitor:///var/log/custom/apache2/*]
ignoreOlderThan=20d
server-input.conf
[monitor:///var/log/custom/report/report*]
sourcetype=report
ignoreOlderThan=20d
I started the forwarder, then made sure the configuration files were downloaded and applied correctly. The log file parses the monitors, but then they don't seem to analyze anything besides the first two sections in the global-inputs file.
Here's splunkd.log:
<snip>
08-10-2012 17:04:19.096 -0400 INFO TailingProcessor - TailWatcher initializing...
08-10-2012 17:04:19.097 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages*.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/syslog*.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/apache2/*.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/backup/*.
08-10-2012 17:04:19.098 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/report/report*.
08-10-2012 17:04:19.099 -0400 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/custom/startup/*.
08-10-2012 17:04:19.099 -0400 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
08-10-2012 17:04:19.103 -0400 INFO TcpOutputProc - Connected to idx=server_address:9578
08-10-2012 17:04:19.124 -0400 WARN TailingProcessor - Insufficient permissions to read file='/opt/splunkforwarder/var/log/splunk/.splunkd.log.swp' (hint: Permission denied).
08-10-2012 17:04:19.126 -0400 INFO ArchiveProcessor - handling file=/var/log/syslog.2.gz
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.126 -0400 ERROR TailingProcessor - matching /var/log/exim4/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.126 -0400 INFO ArchiveProcessor - reading path=/var/log/syslog.2.gz (seek=0 len=8676)
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.128 -0400 ERROR TailingProcessor - matching /var/log/fsck/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.138 -0400 ERROR TailingProcessor - matching /var/log/news/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/apt/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/messages[^/]*$
08-10-2012 17:04:19.139 -0400 ERROR TailingProcessor - matching /var/log/custom/ against ^/var/log/syslog[^/]*$
08-10-2012 17:04:19.144 -0400 INFO ArchiveProcessor - Finished processing file '/var/log/syslog.2.gz', removing from stats
</snip>
Nothing else is entered in the log for a good while after this. The metrics log continues to show connections to the main server.
I've made sure that the splunk user has the correct read permissions on the log files. I'm not getting bad permission errors. It seem to be skipping the other files completely. There's also entries in all the files newer than 20 days (limiting information during testing). The stateOnClient is enabled for each section in the serverclass.conf file.
What should I look for next?
... View more