We are logging OS events with the Nix app on our universal forwarders. As you can see from the attached output, the events are from the exact same host, on the same indexer, from the exact same context, but the timestamps are all wildly different, some of which are set into the future by quite a few years... I've sshed into the box in question and the system clock is correct and the filesystem where these binaries are all correct.
Ive tried setting the MAX_DAYS_HENCE and MAX_DAYS_AGO to 10,000,000, but i still get these warnings. We've considered filtering this log to FATAL level and above, but we'd rather get other WARN level logs if they occur...
Any insight would be greatly appreciated.
tail -f /opt/splunk/var/log/splunk/splunkd.log | grep someHostName
03-12-2012 22:01:30.307 +0000 WARN DateParserVerbose - Accepted time (Mon Jan 18 01:54:00 2010) is suspiciously far away from the previous event's time (Fri Dec 10 02:23:00 2010), but still accepted because it was extracted by the same pattern. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.307 +0000 WARN DateParserVerbose - Accepted time (Tue Jan 26 02:04:00 2010) is suspiciously far away from the previous event's time (Sun Jan 10 01:33:00 2010), but still accepted because it was extracted by the same pattern. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.307 +0000 WARN DateParserVerbose - Accepted time (Sat Jan 30 04:13:00 2010) is suspiciously far away from the previous event's time (Mon Jan 18 01:54:00 2010), but still accepted because it was extracted by the same pattern. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.308 +0000 WARN DateParserVerbose - Accepted time (Fri Jul 15 02:21:00 2016) is suspiciously far away from the previous event's time (Tue Jan 26 02:04:00 2010), but still accepted because it was extracted by the same pattern. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.308 +0000 WARN DateParserVerbose - Accepted time (Mon May 29 12:18:00 2017) is suspiciously far away from the previous event's time (Sat Jan 30 04:13:00 2010), but still accepted because it was extracted by the same pattern. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.308 +0000 WARN DateParserVerbose - A possible timestamp match (Fri Jul 15 02:21:00 2016) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.308 +0000 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event. Context="source::top|host::someHostName|top|remoteport::31870" Text=" 16197 root 15 0 10104 772 616 S 0.0 0.0 2:21.15 syslog..."
03-12-2012 22:01:30.308 +0000 WARN DateParserVerbose - A possible timestamp match ( 17153 ) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::top|host::someHostName|top|remoteport::31870"
03-12-2012 22:01:30.309 +0000 WARN DateParserVerbose - A possible timestamp match ( 17149 ) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context="source::top|host::someHostName|top|remoteport::31870"
... View more