Parsing Date/Time questions (biggest issue is there is no year specified)

[Chris_Olson]

Looking for some guidance on non-standard date/time parsing…

We have a customer that has logs without years

We're going round and round trying to get them to put the year into their logs, but are still getting some pressure to make it work as is

I've looked at using strptime via TIME_FORMAT in props.conf, but this raises some questions

(1) How to account for the comma then millisecs? - I stripped it in the outside tests (subseconds, possibly %3N?)
(2) Strptime testing outside of splunk sets year to 1900 if not specified
(3) Can the year be set to a default or current value in props or transforms?

A sample line is > 1800 characters

when tested manually against a pattern, splunk seems to parse the date OK and substitute the current year for the missing year

when tested against a source file, with lines beginning with the same values, splunk fails

[splunk@box1 samples]$ splunk test dates applog.log
Using logging configuration at /opt/instance/splunk/etc/log-cmdline.cfg.
Unable to parse 'applog.log'
[splunk@box1 samples]$ splunk test dates "23:59:48,243 01/11 INFO aaaStateLogger"
Using logging configuration at /opt/instance/splunk/etc/log-cmdline.cfg.
From: 23:59:48,243 01/11 INFO aaaStateLogger
Parsed: Tue Jan 11 23:59:48 2011
UTC Time: 1294808388
Time Region: 0-13
Date Region: 13-18
Subseconds: 0.243
[splunk@box1 samples]$

I have posted a partial extract of a few lines below

Sample trimmed data:

23:57:44,491 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:57:55,459 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:57:57,233 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:57:57,730 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:58:07,700 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:58:10,515 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:58:50,544 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:59:07,033 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:59:16,737 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

23:59:48,243 01/11 INFO aaaStateLogger - aaaTerminationRequest[aaaState=,menuCode...

[Lowell]

I'm not 100% sure what all you've tried. I would start with simply setting up an appropriate props.conf entry, for example:

[the_source_type]
TIME_FORMAT = %H:%M:%S,%3N %m/%d
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 40
# Assuming your files are on a single line, so you should set this too:
SHOULD_LINEMERGE = False

Do you regularly load these files from the previous year? Or are you concerned about when the year rolls over? If you have splunk actively monitoring your files (either locally, via splunk forwarder, or over a network share) and aren't doing any kind of batching process (or log file shipping), then you should be fine as splunk read the files pretty quickly.

Having said that, I haven't really tested this configuration too closely, as splunk seem to just get it right. That, and the year change doesn't happen all that often...