Splunk is showing the following WARN messages in the logs suddenly, but nothing was changed on the logs or props.conf. I feel the time spent to parse has increased, as many of our scheduled reports have slowed down and search is also affected.
07-29-2015 13:42:27.889 -0700 WARN DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Sun Jul 26 23:48:00 2015). Context: source::/server_logs-ep9/client/zDefaultInstance/12.123.123.123/EAppServer.ESERVER1234.2015-07-26-00.log|host::ehoat001|encompass|1549379
Log Filename - EAppServer.ESERVER1234.2015-07-26-00.log
I checked the logs but the time stamps look to be intact... not sure where we are breaking ..
Example of normal logs:
[07/27/15 13:00:14.0687] APITRACE {057}: <f0109337-1c69-45e7-a4c3-feeed7996650>OrganizationManager.GetScopedUsersWithRole(19) by user/Encompass/Plugin:GenPlugin/123-1234-5678-abcd-1238. ,<546ms>
[07/27/15 13:00:14.6147] APITRACE {057}: <2ae1cb31-649e-468d-ad89-c8e4487df206>CursorBase.GetItemCount() by user/Encompass/Plugin:Main/123-1234-5678-abcd-123. ,<0ms> Example from logs which
Props.conf:
TIME_PREFIX = ^\[
TIME_FORMAT = %m/%d/%y %H:%M:%S.%4N %p
MAX_TIMESTAMP_LOOKAHEAD = 40
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)\[\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+\s*\w*\]
TRUNCATE = 20000
KV_MODE = none
transforms.conf
[enc_set_sourcetype_api]
REGEX = ^[^\]\n]*\]\s+APITRACE
FORMAT = sourcetype::encompass-apitrace
DEST_KEY = MetaData:Sourcetype
You should use this instead (change %4N
to %3N
and %p
to %z
😞
TIME_FORMAT = %m/%d/%y %H:%M:%S.%3N %z
Just insert this in your props.conf sourcetype section:
DATETIME_CONFIG = CURRENT
This will eliminate the error at the expense of ignoring and supplanting every event's timestamp. This is a terrible way to handle it; it will "work" but it will create other (invisible) problems.
Your %p
is looking for AM
and PM
which is not in your logs.
I guess , I did , Not sure what I am missing here.??
I did have a entry in the props.conf to point the datetime.xml . And also in the datetime.xml have the Regex.
Can you help me point out the missing part?
What exactly did you put in props.conf
? As far as datetime.xml
. I has to be in exactly the right spot on the Indexers and it should look something like this:
<datetime>
<define name="_withOrWithoutAMPM" extract="month, day, year, hour, minute, second, subsecond, ampm">
<text>\[(\d{4})/(\d{2})/(\d{2})\s+(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s*(.{0,2})\]</text>
</define>
<timePatterns>
<use name="_withOrWithoutAMPM"/>
</timePatterns>
<datePatterns>
<use name="_withOrWithoutAMPM"/>
</datePatterns>
</datetime>
thanks again :
the following is my props.conf used when I pointed out to datetime.xml
[encompass]
DATE_TIME CONFIG = /etc/apps/EM_encompass_indexer/local/datetime.xml
LINE_BREAKER = ([\n\r])+(?=(\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]|[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]))
SHOULD_LINEMERGE = false
TRUNCATE = 20000
KV_MODE = none
TRANSFORMS-enc_set_sourcetypes = enc_set_sourcetype_api, enc_set_sourcetype_err, enc_set_sourcetype_sql, enc_set_sourcetype_dbg, enc_set_sourcetype_warn, enc_set_sourcetype_info
OK, so if you use this location and put my guts for datetime.xml
on all of your Splunk Indexers and restart all of their Splunk instances, it should work:
$SPLUNK_HOME/etc/apps/EM_encompass_indexer/local/datetime.xml
Thanks . One question would be, what should I put in props.conf for the LINE_BREAKER = ?
Leave it out entirely (let it default).
Tried it out. It is parsing correctly now, but It is not breaking the line at every time-stamp 😞
Sorry, try this:
LINE_BREAKER = ([\n\r]+)\[\d(2)\/\d{2}\/\d{4}\s+\d+\:\d{2|\:\d{2}
@nmohammed You really should come back and update your question. I assume that you got it all working so you should click Accept
to close it out.
cat datetime.xml
\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]
\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]
OK, I see the problem now! Did you read the whole article? This is only 5% of what the file should contain!
Thanks Woodcock,
I created a datetime.xml for my environment, but I guess it has some issues and taking effect:
\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\s(\w{2})\]
\[(\d{2})/(\d{2})/(\d{2})\s\s(\d{1,2}):(\d{2}):(\d{2})\.(\d{4})\]
What is the complete contents of datetime.xml
?
Thanks , some of our client logs do have the localization for AM/PM , because of different version of product
This was not happening earlier and started just few days ago. Can we change the props.conf to index both of these formats ?