I'm working with Splunk Universal Forwarder 6.5.2 and am trying to configure a monitor on the docker volumes directory on docker hosts.
I have a app configuration "inputs.conf" like this:
[default]
host=THEHOSTNAMEHERE
[monitor:///var/lib/docker/volumes/splunk-log-store/.*]
sourcetype=json
index=application
In the splunk-log-store directory I have a number of .log files with JSON content.
[root@THEHOSTNAMEHERE ec2-user]# tail -n 100 $SPLUNK_HOME/var/log/splunk/splunkd.log | grep -E 'permission|docker'
05-02-2019 14:19:35.718 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/lib/docker/volumes/splunk-log-store/.*.
05-02-2019 14:19:35.719 +0000 INFO TailingProcessor - Adding watch on path: /var/lib/docker/volumes/splunk-log-store.
05-02-2019 14:19:35.719 +0000 WARN FilesystemChangeWatcher - error getting attributes of path "/var/lib/docker/volumes/splunk-log-store": Permission denied
05-02-2019 14:19:35.721 +0000 WARN TailReader - Insufficient permissions to read file='/var/log/cron' (hint: Permission denied , UID: 501, GID: 501).
05-02-2019 14:19:35.722 +0000 WARN TailReader - Insufficient permissions to read file='/var/log/secure' (hint: Permission denied , UID: 501, GID: 501).
05-02-2019 14:19:35.722 +0000 WARN TailReader - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied , UID: 501, GID: 501).
However, as per above, I'm getting permission errors when the splunk service starts.
It's running as user "splunk" (UID 501).
This is the particular error around the directory I'm trying to monitor:
WARN FilesystemChangeWatcher - error getting attributes of path "/var/lib/docker/volumes/splunk-log-store": Permission denied
I've tried changing the group permissions on the directory to the splunk group and also tried setting full permissions (completely open) as a hacky test but each time I restart the splunk service I still get this permission error.
Any ideas what I'm doing wrong here?
I have been able to successfully get syslog logs working by modifying permissions on the /var/log/messages etc... but the same modifications for this docker volume directory just don't seem to work.
Thanks
... View more