I have two sourcetypes where the thousandth of a second portion of the timestamp is not padded w/ leading zeros if the time is less than 100 thousandths of a second.
Examples:
A log event is created at eight AM and eight thousandths of a second. The timestamp for this event would be...
08:00:00:8
Splunk interprets that time as...
08:00:00:800
instead of
08:00:00:008
A log event is created at nine AM and ninety thousandths of a second. The timestamp for this event would be...
09:00:00:90
Splunk interprets that time as...
09:00:00:900
instead of
09:00:00:090
This causes log entries in Splunk to be out of order when viewing a sequence of logs.
My question is how I should go about fixing the timestamp for future logs (w/o a huge burden on the indexer). And can I fix the timestamps for events that already exist in my index?
I will include some example logs for additional clarification. Note that all items (including date and time) are tab delimited.
SERVER-1 Aug 05, 2010 12:58:58:851 1234 <SYSTEM/AUTOMATION> Running Job ID 1, Name of job 1
SERVER-1 Aug 05, 2010 12:58:59:2 1235 <SYSTEM/AUTOMATION> Running Job ID 5, Name of job 2
SERVER-1 Aug 05, 2010 12:58:59:70 1235 <SYSTEM/AUTOMATION> Running Job ID 3, Name of job 3
SERVER-1 Aug 05, 2010 12:58:59:132 1235 <SYSTEM/AUTOMATION> Running Job ID 2, Name of job 4
Per Lowell's suggestions, I have tried the following permutations for the time format of my sourcetype in the local props.conf file with no improvement...
TIME_FORMAT = %b %d, %Y"\t"%H:%M:%S:%3N
TIME_FORMAT = %b %d, %Y\t%H:%M:%S:%3N
TIME_FORMAT = %b %d, %Y %H:%M:%S:%3N
TIME_FORMAT = %b %d, %Y %H:%M:%S:%3N
TIME_FORMAT = %b %d, %Y"\t"%H:%M:%S:%Q
TIME_FORMAT = %b %d, %Y\t%H:%M:%S:%Q
TIME_FORMAT = %b %d, %Y %H:%M:%S:%Q
TIME_FORMAT = %b %d, %Y %H:%M:%S:%Q
It's hard to see, but the third and seventh lines have a literal tab between the year and hour, and the fourth and eighth line have a space between the year and hour.
... View more