Deployment Architecture

Universal Forwarder with logrotate - events only arrive once a week

mprilop
Explorer

Hi,
i am having problems troubleshooting missing log entries. The UF seems to be configured correctly (list monitor and splunkd.log indicate everything okay). But only once a week four events reach splunk (it should be several thousand per day). Once per week may point to logrotate which is done weekly (config below) but i could not find any reference to similar problems with current versions of UF and splunk.

Our configuration:
Splunk version 6.1.3 (some enterprise license)

Universal Forwarder

# cat /opt/splunkforwarder/etc/splunk.version 
VERSION=6.1.3
BUILD=220630
PRODUCT=splunk
PLATFORM=Linux-x86_64

monitoring entry:

[monitor:///var/www/*/shared/log/*.log]
disabled = false
sourcetype = rails_app

splunkd.log entry for above monitoring:

01-14-2015 10:36:23.714 +0100 INFO  WatchedFile - Will begin reading at offset=72355036 for file='/var/www/our_app/shared/log/production.log'.

logrotate setting:

/var/www/*/shared/log/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    copytruncate
}
0 Karma
1 Solution

mprilop
Explorer

Turns out i had insufficient rights to view the resulting logs.

I do not know why i still saw single entries and combined with me being able to see all other logs directed to splunk via syslog it never occurred to me to check my access rights with our hoster.

View solution in original post

0 Karma

mprilop
Explorer

Turns out i had insufficient rights to view the resulting logs.

I do not know why i still saw single entries and combined with me being able to see all other logs directed to splunk via syslog it never occurred to me to check my access rights with our hoster.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi mprilop,

take a look at this answer http://answers.splunk.com/answers/185453/why-copytruncate-logrotate-does-not-play-well-with.html provided by @yannK, it includes some work around as well.

cheers, MuS

mprilop
Explorer

Thanks for your suggestion. I will try out the workaround and hope it also solves my problem.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...