Given this timestamp:
01/Mar/2011:17:25:49.666+0000
What is the right format?
I'm leaning towards:
TIME_FORMAT = %d/%b/%Y:$H:%M:%S.%3N+%z
but I'm not quite sure what this means from the extended strptime documentation:
%z, %::z, %:::z GNU libc support.
Unfortunately we don't ship our "DateTimeTest" utility that allows easy testing of strptime/strftime functionality, please file an ER for that to be included from the CLI/UI. Here's my findings (first note that you have $H rather than %H):
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N+%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666+-0800
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666-0800
So basically, %z does essentially the right thing, but the +
is unnecessary and actually causes wrong results.
As a clarification, the :
in %z
says how to split the time zone offset. One :
looks like -08:00
for PST, two :
looks like -08:00:00
.
After reading about %z on these pages and http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Configuretimestamprecognition , I found the python 3 documentation https://docs.python.org/3/library/datetime.html has this explanation:
"
%z UTC offset in the form +HHMM or -HHMM (empty string if the object is naive). (empty), +0000, -0400, +1030 (6)
%Z Time zone name (empty string if the object is naive). (empty), UTC, EST, CST
"
The documentation has been prompt updated to reflect this information correctly!
Furthermore, the preview functionality mentioned is now part of the "add data"/data upload option in Splunk...
%z is -0400 This format is not standard. if your machine is configure as Eastern Date Time
%Z is EDT if your machine is configure as Eastern Date Time, not too much use for storing it in data base. By the way I live in New York.
%:z is -04:00 That is the one most useful in hours and minutes. It can be used across computer languages
%::z is -04:00:00 It is over kill. we don't need second for time zone
%:::z is even more over kill, no use in reality
Actually, you should use the new preview feature (shipped in 4.3) to test any tz configs.
Actually, that's true. Though actually, it was posted 2 years later.
Preview is great for manipulation & validation of timestamp extraction before implementation.
Unfortunately we don't ship our "DateTimeTest" utility that allows easy testing of strptime/strftime functionality, please file an ER for that to be included from the CLI/UI. Here's my findings (first note that you have $H rather than %H):
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N+%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666+-0800
[ssorkin@MrT current]$ src/util/tests/DateTimeTest '%d/%b/%Y:%H:%M:%S.%3N%z' '01/Mar/2011:17:25:49.666+0000'
01/Mar/2011:09:25:49.666-0800
So basically, %z does essentially the right thing, but the +
is unnecessary and actually causes wrong results.
As a clarification, the :
in %z
says how to split the time zone offset. One :
looks like -08:00
for PST, two :
looks like -08:00:00
.
I made a request to ship your tool here:
https://ideas.splunk.com/ideas/EID-I-59
| makeresults
| eval time1="01/Mar/2011:17:25:49.666+0000"
| appendpipe
[ eval time2="2015-10-22T22:41:52.546249+00:00"]
| eval _time=coalesce(strptime(time2,"%FT%T.%6Q%:z"),strptime(time1,"%d/%B/%Y:%T.%3Q%z"))
on Splunk ver8.0.1
btool [options] {check|validate-strptime|validate-regex}
Is this not enough?
Rather than submitting yet another question about timestamp formating, could you please update your answer with the correct format string to successfully parse a timestamp like 2015-10-22T22:41:52.546249+00:00
? I tried %Y-%m-%dT%H:%M:%S.%6N%:z
, but it doesn't seems to be working. The timezone info is probably not being captured as all event times are being translated as if the event timezone (always UTC) was the same as the splunk server (-0500), which, translated to my splunk user timezone (-0200), gives me 2015-10-23T01:41:52.546249-02:00
. Any clue?
Perfect. Thanks.