Stop times like '0:20:00' being read as 8pm

[parallaxed]

Splunk always seems to get this wrong. I have the following in a vain effort to correct this

TIME_PREFIX=^

TIME_FORMAT=%D%t%T

Didn't really do anything for the situation. Wondering if there's some other config I can try?

[parallaxed · Accepted solution]

This is due to the fact that default etc/datetime.xml regex is not greedy enough:

<define name="_hour" extract="hour"> <text><![CDATA[([01]?[1-9]|[012][0-3])(?!\d)]]></text> </define>

Adding an additional clause fixed this:

<define name="_hour" extract="hour"> <text><![CDATA[([01]|[01]?[1-9]|[012][0-3])(?!\d)]]></text> </define>

[gkanapathy]

In addition to Mick's answer (which is that you should use the correct TIME_FORMAT according to what is in your data) you may also need to look into the MAX_TIMESTAMP_LOOKAHEAD setting, which will control how far into a line Splunk will keep looking for a timestamp. Simply setting the TIME_PREFIX just tells Splunk to start looking for times is sees the TIME_PREFIX, but it could be anywhere after the prefix.

If you provide what an actual line of data looks like, perhaps we can help. If the TIME_FORMAT can't be found, Splunk rather aggressively attempts to derive some timestamp from your data, as a time is always required to write an event into the Splunk index.

[gkanapathy]

i'd say pretty much a bug. it might be possible to work around it with a custom datetime.xml setting.

[parallaxed]

I've tried both, neither work.

[gkanapathy]

Could you try using %H:%M:%S instead of %T? I have never used %T.

[parallaxed]

Additionally, as a consequence of the above, the times 0:01:00 through 0:23:00 are also affected (interpreted as 24 hour times exc. leading 0). 0:24:00 is recognized correctly as 12:24AM, example line:

05/20/2010 0:24:00,someData,1274329440,60,0,2,8256952.0,3913828.0,4007580.0,50.0

[parallaxed]

Further to the comment above, I should add that I've tested with %D%t%t%t%T and more specific strptime formats, none of them seem to work with this kind of event (i.e. missing a leading 0 off the %T). Is this a bug?

[parallaxed]

Seems whitespace doesn't work, hope that's clear though...

[parallaxed]

The event line looks like the following:

05/20/2010 0:20:00,someData,1274329200,60,0,2,someData,8256952.0,3913828.0,4007580.0,50.0

^ the timestamp is recognized as 8pm.

The only exception that I can see here is that there are three lots of whitespace between %Y and %H, shouldn't %t cover multiple spaces anyhow? It seems the '0:' following %t is being ignored.

[Mick]

There's many other possible configs you can try, but trying to guess the exact format of the entire timestamp is beyond my capabilities. When you ask a question that'd dependant on the format of an event, why not paste in an example so we can see what you're talking about? If there's sensitve data in there just anonymize it with x's or something.

This is the page you want to read first - Enhanced strptime() support and then here

Your setting - TIME_FORMAT=%D%t%T

%D - The date as %m / %d / %y.
%t - white space
%T - The time as %H : %M : %S.

Where did you get these values?

How about - TIME_FORMAT=%H:%M:%s

%H = The hour (24-hour clock)
%M = The minute [00,59]
%S = The seconds [00,60]

Note the ':' in there as well, you're specifying the exact format of the timestamp so you have to include everything that is contained within it.