Indexing logs in /var/log

[gnovak]

Ok I have been trying to get /var/log/messages and /var/log/cron to be indexed as the "splunk" user for a while and I'm pretty frustrated.

I read this question here:

http://splunk-base.splunk.com/answers/60388/recommended-permissions-on-varlog-for-splunk_ta_nix

I followed some of the advice here and set permissions for /var/log directory using ACLs. I also tried making the splunk user a member of the adm group. I EVEN got to the point of changing permissions on the file itself with chmod and the file still did not show up as being indexed. I made sure I restarted splunk each time on the forwarder when I made these changes.

I'm running Centos 6.2 Has anyone else had this issue? I could use UDP for port 514 but this way seemed a bit more sense and less config changes.

The error i found in splunkd.log for this is:

02-07-2013 13:41:29.441 -0500 WARN FilesystemChangeWatcher - error getting attributes of path "/var/log/messages": Permission denied

Obviously these changes aren't making a difference. Anyone else have this problem? Someone mentioned to me today "selinux" could be an issue. I also thought about giving sudo all access for the splunk user. Anyone else try this?

[crash1011]

In case you get trapped with a file not being monitored even if (1) all permissions seem correct, (2) your deployment script is set to Enable App, Restart Splunkd and (3) You see these errors
09-18-2015 12:28:47.311 +1000 WARN FilesystemChangeWatcher - error getting attributes of path "/software/app/oracle/admin/webhost1/diagnostics/logs/OHS/ohs1/access_log": Permission denied
Then I found this actually did work:
- Log on to the forwarder and check that your app with the file monitoring stanza has been deployed all OK
- Do a splunk list monitor (if you’ve got the same problem it won’t be listed)
- Restart of splunk e.g. /opt/splunkforwarder/bin/splunk restart
- Do another splunk list monitor to see if it has worked

Unfortunately in this exercise I didn’t do a ps | grep splunk on the remote host to check if the splunkforwarder process had been restarted by the utility server’s splunk reload deploy-server

[tfpblanchard]

I would suggest it's a problem with SELinux.
Try to disable it temporarily:

[root@server ~]# getenforce
Enforcing
[root@server ~]# setenforce Permissive
[root@server ~]# getenforce
Permissive

It's not recommended to disable SELinux unless you know what you're doing, so check out Splunk on SELinux

[gnovak]

I eventually did remove the acl permissions and just set new ones w/ chmod. It seemed the acl permissions were clashing with others on the file. I added splunk to an admin group and changed the owner of the file to be root:admin. This finally worked. However If I could not make all of these changes because of security, is there another way...