I am trying to index a WebSphere WAS log, where the time sone format is like this
[24/11/11 10:49:57:538 EST] 0000004a ServletWrapper I SRVE0242I: [custom-webapp] [/app]
What I actually want Splunk to do is treat this date time as Australian 'as is'
- namely the date in the index = exactly the same as the date recorded
I've tried the following changes to props.conf without success
[host::my_host*]
TZ = Australia/Melbourne
I suspect that Splunk sees EST in the input file and assumes its US EST
and then sees TZ and adds the diff between US EST and Oz to the input values
- thus winding my logs entries fwd
I just want them to be treated 'as is'
Can you help ?
I don't think the props.conf gets totally bypassed, this doc, mentions that Splunk does look for the date/time in the event, but using the TIME_FORMAT
(if present in props.conf).
Have tried defining the TIME_FORMAT
in props.conf? - I was also thinking about limiting the "lookahead" to limit how far Splunk looks for the date/time (i.e. stopping it before it gets to the "EST"), for this you would use something like "MAX_TIMESTAMP_LOOKAHEAD = 22
" (which should stop the lookahead before the EST).
An example of this can be found here for the TIME_FORMAT=<strptime>
. And here for an example using MAX_TIMESTAMP_LOOKAHEAD
.
Also to ask an obvious question are you using the "$SPLUNK_HOME/bin/splunk clean
", to remove the old incorrect events?
Still trying to get this to work properly. The problem with TZ_ALIAS is that it doesn't work on the right part of the problem. Sure you can change the TZ but WAS has already inserted the wrong timestamp into the event log (14 hours off) so it doesn't help. What is needed to is to rewrite the timestamp in the event (EST) using the offset to the correct timestamp (system time, in this case AEST) so that they agree.
Anyone got a recipe for this? Don't want to reinvent the wheel if I don't have to. Alternatively I'm thinking I might be able to simply discard the time in the event, or perhaps extract it as a field and then replace it with _indextime to hide the wrong values. Ugly, and should not be necessary.
Or IBM could make system time and not Armonk time the default in WAS. Yeah, and I will keep a lookout for articles about porcine aviation in the newspapers 🙂
cm
Yes and no minister. I tried TZ_ALIAS yesterday and nothing changed. Can we please have some more detail on how to deploy this change, for those of us who do not have a WAS installation to mess around with.
Does it need to go in the UFs? In the appliance? On the indexer? Can it be set globally or do we have to edit each and every source or sourcetype that the WAS app uses? And when is the solution going to be built in to the app?? Looks like this problem has been around for quite a while.
Charles
As of Splunk 5.0.2 this problem has been resolved!
You can now set what timezone offset an acronym like "EST" refers to - in props.conf
http://docs.splunk.com/Documentation/Splunk/latest/Data/ApplyTimezoneOffsetsToTimeStamps
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
(Search the props.conf
manual page for "Australia" and you'll find the right section)
This needs to be added to the documentation as a 'gotcha'. WSA uses EST UNITED STATES by default if you don't set a time zone in the WS environment variables. It's NOT EST Australia. Changing this value once WSA apps are deployed may create unknown side effects, according to the websphere guys I'm talking to.
Note that the end result of this for diligent system admins who've set up ntp is that the time zone in the events does not agree with the system time that Splunk ingests the event on. This causes a lot of interesting problems in the app, as I have just discovered at my customer site.
Can we please have an official, tested and supported solution for this problem which does not involve messing around with WSA, as that would typically involve change control, outages, and maybe even app rewrites which might be out of the question.
I tried configuring the props.conf by defining parameter for a particular source, but didn't work. Managed to find 2 other alternatives!
1st solution:
As all my log files are going to be on Australia timezone, simplest solution was to remove CST & EST from datetime.xml. That's it.
2nd solution:
Defined a new source time that ignores CST. Steps to do this:
1.Go to Manager > Data inputs > Files & directories > New
2.Select option 'Preview data before indexing'
3.Select the source type.
4.You would see that the time stamp selected/highlighted would include CST/EST.
5.Select 'adjust timestamp and event break settings'
6.Select 'Timestamps' tab
7.Select option 'Timestamp never extends more than xxx chars into the event', and define the right number so that CST/EST is ignored.
8.After applying your new configuration and verified it ignores CST/EST, select 'Continue' and save it as a new source type.
9.Import logs under new source type and it should work.
Sorted using the tip on
MAX_TIMESTAMP_LOOKAHEAD = 22
Found 2 other alternatives! Posting as an answer.
Need assistance as I think I'm not doing this right. I'm using Splunk 5 for windows.
Here's what my log file looks like: e.g. ####<*Nov 12, 2012 4:04:35 PM CST>
Right now Splunk catches CST and translates into US time, so it shows as 11/13/12 9:00:14.000 AM.
There was no props.conf in my C:\Program Files\Splunk\etc\system\local directory so created one:
[source::*]
TZ = Australia/Adelaide
TIME_PREFIX = ####<
TIME_FORMAT = %b %d, %Y %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 28
I am sure it must have been my TIME_PREFIX & TIME_FORMAT formatting error. Can anyone check and confirm my sytax? Cheers!
Awesome, can you make the answer as accepted please? Thanks.
I don't think the props.conf gets totally bypassed, this doc, mentions that Splunk does look for the date/time in the event, but using the TIME_FORMAT
(if present in props.conf).
Have tried defining the TIME_FORMAT
in props.conf? - I was also thinking about limiting the "lookahead" to limit how far Splunk looks for the date/time (i.e. stopping it before it gets to the "EST"), for this you would use something like "MAX_TIMESTAMP_LOOKAHEAD = 22
" (which should stop the lookahead before the EST).
An example of this can be found here for the TIME_FORMAT=<strptime>
. And here for an example using MAX_TIMESTAMP_LOOKAHEAD
.
Also to ask an obvious question are you using the "$SPLUNK_HOME/bin/splunk clean
", to remove the old incorrect events?
Awesome, can you make the answer as accepted please? Thanks.
Sorted 😉
The change to using MAX_TIMESTAMP_LOOKAHEAD was the key
without it the system always gave preference to the EST and assumed data was from the US EST zone. Now it stores 'as is' in Oz time
I have tried the above without success 😞
By default, Splunk applies time zones using these rules, in this order:
FIRSTLY. Splunk uses any time zone specified in raw event data (for example, PST, -0800).
And my log has an eyecatching EST in it !
e.g. [24/11/11 10:49:57:538 EST] 0000004a ServletWrapper I SRVE0242I: [custom-webapp] [/app]
Then ... Splunk uses the value of a TZ attribute set in props.conf, if the event matches the host, source, or source type specified by the stanza.
And then ... Splunk uses the time zone of the Splunk server that indexes the event.
So I suspect my props.conf is being bypassed because it sees EST in there already.
My host name matches the wildcard in the props stanza
I assume you have restarted the services, I also assume you are using the host name used in the events (either configured in inputs.conf or the default in your server.conf file).
Or maybe you can use "DATETIME_CONFIG = CURRENT" in your props.conf (check http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf for more info), this should assign the event time as the system time. Is this any help?
Regards,
PS: One option I have considered is this
The documentation says it evaluates timezone in this order
a. seeing values in the log like EST
b. then using values in props
c. then using time set on the server
So I think in my case (a) is being applied before it even considers b, hence my mods to props.cong are redundant
So ... how I teach Splunk to recognise Oz style timezones ??