How to handle a logfile with two timestamps

[a212830]

Hi,

I have a catalina.out logfile that starts with one timestamp (YYYY-MM-DD HH:MM:SS), and then, after the initialization is complete, starts using a different one (MM.DD.YYYY HH:MM:SS). The customer needs all events, so I can't filter out the startup. How would this get handled?

2014-11-02 04:15:52 [cf6fb877] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226
2014-11-02 04:15:52 [cf6fb877] info    [native] Using /fidcn/install/dynatrace-5.5.0 as Agent storage
2014-11-02 04:15:52 [cf6fb877] info    [native] Log file at /fidcn/install/dynatrace-5.5.0/log/dt_CTG_FidelityCentral_prod_bootstra
p_4145.log
2014-11-02 04:15:52 [cf6fb877] info    [native] Trying to connect to dtcoll-ctgprod-mmk.fmr.com:9998 for up to 60 seconds
2014-11-02 04:15:52 [cf6fb877] info    [native] Connected successfully
...
03.11.2014 06:22:45 ERROR [org.apache.jetspeed.container.JetspeedContainerServlet.doGet():350] JetspeedContainerServlet: Error rendering portlet "fcapps::FCStats": javax.portlet.PortletException: Request processing failed
javax.portlet.PortletException: Request processing failed
        at org.springframework.web.portlet.FrameworkPortlet.processRequest(FrameworkPortlet.java:544)
        at org.springframework.web.portlet.FrameworkPortlet.doDispatch(FrameworkPortlet.java:470)
        at javax.portlet.GenericPortlet.render(GenericPortlet.java:253)
        at org.apache.jetspeed.portlet.PortletObjectProxy.proxyRender(PortletObjectProxy.java:280)
        at org.apache.jetspeed.portlet.PortletObjectProxy.invoke(PortletObjectProxy.java:172)
        at $Proxy179.render(Unknown Source)
        at org.apache.jetspeed.factory.JetspeedPortletInstance.render(JetspeedPortletInstance.java:117)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.doFilter(JetspeedFilterChain.java:142)
        at org.apache.jetspeed.container.services.JetspeedFilterChain.processFilter(JetspeedFilterChain.java:92)
        at org.apache.jetspeed.container.services.JetspeedFilterManager.processFilter(JetspeedFilterManager.java:112)
        at org.apache.jetspeed.container.JetspeedContainerServlet.doGet(JetspeedContainerServlet.java:290)

[cpetterborg · Accepted solution]

This is not too difficult (after I figured it out) with 4 different timestamps. I handled it in my installation using the following in the props.conf file:

[tomcat_catalina]
DATETIME_CONFIG = /etc/slave-apps/all_tomcat_props/local/catalina.xml
LINE_BREAKER = ^(?:(?:\w+\s*\d{1,2},\s*\d{4}\s*\d{1,2}:\d{2}:\d{2}\s*\w+)|(?:\[\w{1,7}\]\s\d\d/\d\d/\d\d\s)|(?:\d{4}-\d{2}-\d{2}\s\d{1,2}:\d{2}:\d{2},\d{3})|(?:\[\d\d/\d\d/\d\d\s\d\d:\d\d:\d\d[:,.]\d\d\d\]))
TRUNCATE = 100000
NO_BINARY_CHECK=1
MAX_TIMESTAMP_LOOKAHEAD=200
#  Added transform to remove date from sourcename 20140324
TRANSFORMS-tcatdatestrip = tcat_strip_dates

Then I use the following catalina.xml file:

<datetime>
<define name="_mydatetimeformat4" extract="month, day, year, hour, minute, second, subsecond">
    <text>\[(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)\]</text>
</define>
<define name="_mydatetimeformat3" extract="year, month, day, hour, minute, second, subsecond">
     <text>(\d{4})-(\d{2})-(\d{2})\s(\d{1,2}):(\d{2}):(\d{2}).(\d{3})</text>
</define>
<define name="_mydatetimeformat2" extract="year, month, day, hour, minute, second, subsecond">
    <text>(\d\d)/(\d\d)/(\d\d)\s+(\d\d):(\d\d):(\d\d)[:,.](\d\d\d)</text>
</define>
<define name="_mydatetimeformat1" extract="litmonth, day, year, hour, minute, second, ampm">
     <text>(\w+)\s*(\d{1,2}),\s*(\d{4})\s*(\d{1,2}):(\d{2}):(\d{2})\s*(\w+)</text>
</define>
<timePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</timePatterns>
<datePatterns>
     <use name="_mydatetimeformat4"/>
     <use name="_mydatetimeformat1"/>
     <use name="_mydatetimeformat2"/>
     <use name="_mydatetimeformat3"/>
</datePatterns>
</datetime>

And for completeness, the transforms.conf file entry:

[tcat_strip_dates]
# Remove YYYY-MM-DD  style date from the filename
# /opt/TcatServer*/logs/namestuff.2014-02-13.log
#
SOURCE_KEY = MetaData:Source
DEST_KEY = MetaData:Source
REGEX = (/\w{3}/\S+/\w{4}/\w+)\.\d{4}-\d{2}-\d{2}(.log)
FORMAT = source::$1$2

If you don't have the same filenaming problem that I did (the dates in the filename), you can not to the transforms stuff in the props and transforms files.

[cpetterborg]

You have an open `` in the xml:

<datetime>
 <!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
 <define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
 <text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>

 <?-- Right here is the extra <text>: -->
 <text>

 </define>
 <!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
 <define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
 <text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
 </define>
 <timePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </timePatterns>
 <datePatterns>
 <use name="_datetimeformat1"/>
 <use name="_datetimeformat2"/>
 </datePatterns>
 </datetime>

That could be your problem. Not sure about that though. Try it and let us know.

[cpetterborg]

Looks like it didn't put the <text> in my comment previously. Where is has "You have an extra `in the xml," insert thetext` tag between the quotes.

[a212830]

This is essentially what I've done, but I get that strange error. Am I missing something?

[a212830]

I found this blog entry, and gave it whirl, but no luck.

http://www.function1.com/2013/01/oh-no-splunking-log-files-with-multiple-formats-no-problem

Here's my new datetime.xml file:

<datetime>
<!-- 2014-06-11 11:08:50 [4d313890] info    [native] dynaTrace Bootstrap Agent 5.5.0.5226 -->
<define name="_datetimeformat1" extract="year, month, day, hour, minute, second">
<text>(\d{4})-(\d{2})-(\d{2})\s(\d{2}):(\d{2}):(\d{2})</text>
<text>
</define>
<!-- 11.06.2014 11:12:47 ERROR [org.springframework.web.portlet.FrameworkPortlet.processRequest():559] Could not complete request -->
<define name="_datetimeformat2" extract "day, month, year, hour, minute, second">
<text>(\d{2})\.(\d{2})\.(\d{4})\s(\d{2}):(\d{2}):(\d{2})</text>
</define>
<timePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</timePatterns>
<datePatterns>
<use name="_datetimeformat1"/>
<use name="_datetimeformat2"/>
</datePatterns>
</datetime>

Here's my props.conf, using the data preview:
DATETIME_CONFIG = /etc/apps/SVB/datetime.xml
LINE_BREAKER LINE_BREAKER = ([\r\n]+)\d{4}\d{2}\d{2}\s\d{d}:\d{2}:\d{2([\r\n]+)\d{2}.\d{2}.\d{4}\s\d{2}:\d{2}:\d{2}
SHOULD_LINEMERGE=false

When I try it, I get:
Generated an empty preview: check your line breaking and line merging settings; Uncaught exception in Aggregator, skipping an event: Error parsing regex XML file: /apps/splunk/etc/apps/SVB/datetime.xml - Couldn't find 'timePatterns' in config data for AggregatorProcessor.