log unsafe as it does not exist anymore, scheduling a oneshot timeout instead.

[kmarciniak]

Running syslog-ng with a HF. Logrotate runs hourly. 16 or so different web proxies are sending logs to the syslog-ng server with the HF. Sometimes 1 out of the 16 proxy log sources are no longer getting read by the HF even though the proxy log file exists in syslog-ng and can be read. At the top of the hour it fixes itself and the HF reads the file but i'm out of logs for an hour for correlation rules. DEBUG was enabled and below shows the log right at the time of the last log event seen in Splunk

01-25-2019 22:00:05.997 +0000 DEBUG TailingProcessor - Defering file=/var/syslog/proxy/192.168.251.141/proxy.log unsafe as it does not exist anymore, scheduling a oneshot timeout instead.

./splunk list inputstatus | grep -A4 proxy | grep -A4 192.168.251.141
/var/syslog/proxy/192.168.251.141
parent = /var/syslog/proxy//.log
type = directory
<<<<<<<>>>>>>>
/var/syslog/proxy/192.168.251.141/proxy.log
file position = 30610690
file size = 30610690
parent = /var/syslog/proxy//.log
percent = 100.00
I read logrotate should not be used with syslog-ng. Anyone ever see this message?

[jkat54]

You need to put some variables in your syslog-ng destination.

Using just proxy.log is causing a racing condition

2019-01-21-1900-proxy.log would avoid this (assuming you logrotate hourly).

Month, day, year, hour is usually good for most data sources. Larger data sources, we typically add hour to as well.

There are severa macros for acheiving this in syslog-no. Please refer to the syslog-ng documentation for these macros.

[jkat54]

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-...

see the link above.

Note C_HOUR is the Current Hour in which the data arrived whereas HOUR is what syslog-ng extracted. For that reason we always use the “Current” or “C_” macros in file paths and names. This is so that if you get an event from 1982 it goes into the file named after the current time stamp and not 1982.

Splunk should then be able to extract the actual date of the event without having “seen this file before (crcinit issue)” AND logrotate will function flawlessly.

[woodcock]

I use logrotate with syslog-ng all the time. You should open a support caes.

[kmarciniak]

note: the<<<<>>>> should have said the below output is not seen during the time of the problem, there is no log percent=100, that line is just blank