Should it really be like this? I think it is a bug.
In /var/log I have lots of files and dirs.
I want to monitor the files in /var/log, but I don't want to recurse through all the subdirs. I'm only interested in a few specific subfolders.
In inputs.conf of my universal forwarder (on a Debian machine) I have stanzas like this:
### Files in /var/log
[monitor:///var/log]
disabled = false
index = foo
sourcetype = syslog
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d|/all.log|/faillog|/lastlog|/wtmp)$
### Apache logs
[monitor:///var/log/apache2]
disabled = false
index = foo
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d)$
### qqq logs
[monitor:///var/log/qqq]
disabled = false
index = foo
sourcetype = qqq
ignoreOlderThan = 14d
recursive = false
blacklist = (\.gz|\.\d)$
Bug/problem:
Only the files in /var/log (such as /var/log/messages) get indexed! The UF is not starting to monitor the apache2 nor qqq directories. 😞
Apparently the option "recursive = false" in the first stanza for /var/log is not constrained to that stanza, but also put restrictions on all other stanzas related to the /var/log path.
If I comment out the first stanza (or its option "recursive = false"), the other stanzas start working.
Have I missed some vital keyword/option in the manual, 'cause this can't be how Splunk is supposed to behave, is it?
(both Debian and the UF use the latest version)
Not a solution to this bug, but a workaround:
I removed all occurrances of the problematic option recursive
.
In its place I did two changes:
* instead of monitoring a dir, /var/log, I monitor the items within the dir, /var/log/*
* I added a blacklist regexp filter to exclude any and all subfolders, /var/log/.*/
Result: I monitor all files in the dir but don't recurse into any subfolders. In my two subfolders of interest I do the same.
Example:
[monitor:///var/log/*]
blacklist = /var/log/.*/
blacklist = (\.gz|\.\d|/all.log|/faillog|/lastlog|/wtmp)$
disabled = false
index = foo
sourcetype = syslog
ignoreOlderThan = 14d
### Apache logs
[monitor:///var/log/apache2/*]
blacklist = /var/log/apache2/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
ignoreOlderThan = 14d
### qqq logs
[monitor:///var/log/qqq/*]
blacklist = /var/log/qqq/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
sourcetype = qqq
ignoreOlderThan = 14d
This question of yours helped me identify an issue which I had no clue of why when I set recursive = false on a sub folder doesn't monitor files in that folder. I've noticed a parent folder setting of recursive = false and that appears to be like an obvious issue.
why to give 'blacklist' of Specific extensions of compressed files to exclude, where splunk ignores
packed_extensions_list:
bz, bz2, tbz, tbz2, Z, gz, tgz, tar, zip
Not a solution to this bug, but a workaround:
I removed all occurrances of the problematic option recursive
.
In its place I did two changes:
* instead of monitoring a dir, /var/log, I monitor the items within the dir, /var/log/*
* I added a blacklist regexp filter to exclude any and all subfolders, /var/log/.*/
Result: I monitor all files in the dir but don't recurse into any subfolders. In my two subfolders of interest I do the same.
Example:
[monitor:///var/log/*]
blacklist = /var/log/.*/
blacklist = (\.gz|\.\d|/all.log|/faillog|/lastlog|/wtmp)$
disabled = false
index = foo
sourcetype = syslog
ignoreOlderThan = 14d
### Apache logs
[monitor:///var/log/apache2/*]
blacklist = /var/log/apache2/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
ignoreOlderThan = 14d
### qqq logs
[monitor:///var/log/qqq/*]
blacklist = /var/log/qqq/.*/
blacklist = (\.gz|\.\d)$
disabled = false
index = foo
sourcetype = qqq
ignoreOlderThan = 14d