revised as requested for better background information
Hi I have a newb time zone question.
What have I configured incorrectly that is preventing splunk from applying the TZ rules defined in props.conf to index UTC time zone files correctly?
I've set up a props.conf file with a rule that defines the servers to default to Canada/Mountain and then specifies UTC time zone for log4j files.
I was going to add a [sourcetype::log4j_appian] stanza to the props.conf but I believe according to the precedence rules described in the manual that the [host::abserver*] stanza will override that value anyway, so I was forced to use the source keyword stanza.
http://www.splunk.com/base/Documentation/latest/Admin/Applytimezoneoffsetstotimestamps
http://www.splunk.com/base/Documentation/latest/admin/Propsconf
Precedence:
For settings that are specified in multiple categories of matching stanzas,
[host::] spec settings override [] spec settings.
Additionally, [source::] and [] settings.
[t807309@abserver-web local]$ cat props.conf
[rule::access_common_vhost]
sourcetype = access_common_vhost
#access_common_vhost: some.virtual.host 204.191.153.144 - -[05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
#access_common: 204.191.153.144 - - [05/May/2010:21:50:01 -0700] "GET /arsys/shared/images/login_image.jpg HTTP/1.1" 200 21617
#MORE_THAN_75 = ^\S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$
MORE_THAN_75 = ^\S+ \S+ \S+ \S+ \[[^\]]+\] "[^"]+" \S+ \S+$
[host::abserver*]
TZ = Canada/Mountain
[source::/appian/logs/*.log]
TZ = UTC
server #1: abserver-eng:
server in Canada/Mountain timezone
has props.conf
index server
server #2: abserver-app:
server in Canada/Mountain timezone
has props.conf
standard forwarder; will become light forwarder
weblogic server (weblogic_stdout)
log4j log files with custom sourcetype (log4j_appian) assigned
[t807309@abserver-app local]$ cat inputs.conf
[monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log]
disabled = false
followTail = 0
index = main
sourcetype = weblogic_stdout
[monitor:///opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log]
disabled = false
followTail = 0
index = main
sourcetype = weblogic_stderr
[monitor:///appian/logs/*.log]
disabled = false
followTail = 0
index = main
sourcetype = log4j_appian
Samples:
/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log
mixed mode;
Weblogic lines have GMT stamp:
log4j format; no TZ stamp; GMT:
2010-04-23 21:08:07,434 [Main Thread] DEBUG com.appiancorp.kougar.mapper.parameters.ArrayParameterConverter - performing item-by-item conversion of return value <[Lcom.appiancorp.suiteapi.process.TypedVariable;@2575e61> to
/opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_error.log
no TZ stamp; looks like GMT
May 5, 2010 9:13:22 PM com.metaparadigm.jsonrpc.JSONRPCBridge registerLocalArgResolver
INFO: registered local arg resolver com.metaparadigm.jsonrpc.JSONRPCBridgeServletArgResolver for local class com.metaparadigm.jsonrpc.JSONRPCBridge with context javax.servlet.http.HttpServletRequest
javax.servlet.ServletException: Could not find the config file: /WEB-INF/decorators.xml
/appian/logs/application-server.log
log4j format; no TZ stamp; GMT:
2010-05-08 01:16:46,393 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.asi.components.grid.internal.GridAction - The Forum you are attempting to interact with has either been deleted or does not exist.
com.appiancorp.asi.components.common.WebComponentException: The Forum you are attempting to interact with has either been deleted or does not exist.
server #3: abserver-web:
server in Canada/Mountain timezone
has props.conf
standard forwarder; will become light forwarder
apache web server (log file is a variation on access_common, with virtual host name prepended to each line and apache_error logs)
[t807309@abserver-web local]$ cat inputs.conf
[monitor:///var/log/httpd/*_error_log_current]
disabled = false
followTail = 0
index = main
sourcetype = apache_error
[monitor:///var/log/httpd/*_access_log_current]
disabled = false
followTail = 0
index = main
sourcetype = access_common_vhost
Samples:
/var/log/httpd/vhost_F5_80_error_log_current
no TZ stamp
[Wed May 05 16:03:58 2010] [error] FAILOVER_REQUIRED [line 483 of ap_proxy.cpp]: Service Unavailable
/var/log/httpd/vhost_F5_80_access_log_current
standard apache time format
[t807309@abserver-web 05-May]$ tail vhost_F5_80_access_log_2010-05-10
abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:00 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:03 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:05 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
abserver-web..internal.domain.name 192.168.170.11 - - [10/May/2010:12:22:08 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
abserver-web..internal.domain.name 192.168.170.12 - - [10/May/2010:12:22:10 -0600] "GET /suite/portal/i18nredirect.jsp" 200 1956
The server TZ is set correctly:
[t807309@abserver-app splunk]$ date
Fri May 7 17:57:42 MDT 2010
Here are two sample lines from each of the log files:
[t807309@abserver-app splunk]$ tail /appian/logs/application-server.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404
[t807309@abserver-app splunk]$ tail /opt/bea/logs/nm/nm-bpm-abserver-app/bpm-11_bpm-app-11/bpm-app-11_output.log
2010-05-07 23:27:21,245 [ExecuteThread: '9' for queue: 'weblogic.kernel.Default'] ERROR com.appiancorp.ap2.PortalResponse - Error: 404
Both of these events are stamped as 11:27:21 pm, date_zone=-360 (MDT)
Here's what I see in splunk:
http://www.freeimagehosting.net/uploads/1583444cc8.gif
The only thing I am doing outside the box is assigning a different sourcetype (log4j_appian) to the /appian/logs/*.log files. When I look at the events, Splunk has correctly parsed the timestamps however, so I assume no further definition is required.
Here's the inputs.conf stanza that defines the appian log files:
[monitor:///appian/logs/*.log]
disabled = false
followTail = 0
index = main
sourcetype = log4j_appian
Do I need to do more in terms of defining the custom sourcetype for Splunk to be able to assign the correct TZ?
What (else) am I doing wrong here?
thanks...
... View more