Events indexed in future when UTC timestamp rolls over

[roller]

Hello all,

I have been having some issues with Splunk indexing events from a particular log with a time in the future. I currently have a universal forwarder setup to push all logs in a particular directory to the Splunk indexer/server. The problem only occurs with one particular log with the following format:

23:45:43.992 - INFO [blah blah blah] - 20130827T034543Z|172.16.1.95|
23:45:44.015 - WARN [blah blah blah] - No login context available, unable to proceed

When the UTC time rolls over to the next day (ie 20130826T235959Z to 20130827T000000Z) the date on the index also changes and starts timestamping the events as happening at 8pm on the next day (we are EDT - both where the logs are coming from and where the indexer is). The time goes back to being indexed correctly once the UTC time hits 20130827T040000Z or basically the hour:minute:second time that is at the beginning of each log line hits 24hr.

I have tried configuring props.conf on the indexer to use only the long UTC string without luck.
Ive tried the following:

TIME_PREFIX = (\d{2}:\d{2}:\d{2}\.\d{3})
TIME_FORMAT = %Y%m%dT%H%M%SZ
TZ = US/Eastern

I think that part of the problem is that not all of the lines in this log have the UTC time string. If I try and pick up only on the time at the beginning, the index still gets thrown off after the UTC time rolls over.

There are separate logs from the same source with only the UTC string at the beginning of each line that have no problems. Only the one with the H:M:S at the beginning seem to have issues.

I hope that this is clear. If not, I can attempt to clarify more.

Thanks in advance,

[kristian_kolb]

well, I assume that the starting H:M:S.ms timestamp exists in all messages - and is correct. If that is the case, then both the TIME_FORMAT and TIME_PREFIX are wrong.

[sourcetype]
TIME_PREFIX = ^
TIME_FORMAT = %H:%M:%S.%3N
MAX_TIMESTAMP_LOOKAHEAD = 15
TZ = US/Eastern

Normally you'd use the sourcetype as a base for your timestamp extraction, so if you monitor multiple logfiles in the same directory in the same stanza, you can only set one sourcetype for them all. So if timestamps differ between log files, this might not be correct.

If you could configure your logging to also include the date on each line, you'd be settled.

/K