Universal Forwarder with logrotate - events only arrive once a week

[mprilop]

Hi,
i am having problems troubleshooting missing log entries. The UF seems to be configured correctly (list monitor and splunkd.log indicate everything okay). But only once a week four events reach splunk (it should be several thousand per day). Once per week may point to logrotate which is done weekly (config below) but i could not find any reference to similar problems with current versions of UF and splunk.

Our configuration:
Splunk version 6.1.3 (some enterprise license)

Universal Forwarder

# cat /opt/splunkforwarder/etc/splunk.version 
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

monitoring entry:

[monitor:///var/www/*/shared/log/*.log]
disabled = false
sourcetype = rails_app

splunkd.log entry for above monitoring:

01-14-2015 10:36:23.714 +0100 INFO  WatchedFile - Will begin reading at offset=72355036 for file='/var/www/our_app/shared/log/production.log'.

logrotate setting:

/var/www/*/shared/log/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    copytruncate
}

[mprilop · Accepted solution]

Turns out i had insufficient rights to view the resulting logs.

I do not know why i still saw single entries and combined with me being able to see all other logs directed to splunk via syslog it never occurred to me to check my access rights with our hoster.

[MuS]

Hi mprilop,

take a look at this answer http://answers.splunk.com/answers/185453/why-copytruncate-logrotate-does-not-play-well-with.html provided by @yannK, it includes some work around as well.

cheers, MuS

[mprilop]

Thanks for your suggestion. I will try out the workaround and hope it also solves my problem.