dbconnect2 timestamp parsing

[robertlabrie]

DBX2. Latest versions of both (as of today). Wn7 x64. Data is in MySQL.

timestamp="2007-10-02 20:31:47"

That's an actual timestamp in my data. Field is a varchar.

timestamp format in the input config is: yyyy-MM-dd HH:mm:ss

The value splunk got for _time 2015-09-19T20:31:47.000-04:00

OUtput timestamp format is: YYYY-MM-dd HH:mm:ss
I also tried that format in the input config.'

The only thing that gets parsed reliably is the minutes and seconds. Everything else is wrong. DBX2 is 1000x better than DBx1. Easy to use, looks great. Why is time parsing so hard? Is it not SimpleDateFormat? Is it python? Do I need to use the percents?

[woodcock · Accepted solution]

Check for a log like the following in splunkd.log:

mm-dd-yyyy HH:MM:SS.### +0000 WARN  DateParserVerbose - A possible timestamp match (--------------------) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.  Context="source::--------------------"

From the props.conf docs:

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, that an extracted date can be valid.
* For example, if MAX_DAYS_AGO = 10, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000 (days), maximum 10951.
* IMPORTANT: If your data is older than 2000 days, increase this setting.

So, set MAX_DAYS_AGO to something high enough in props.conf and you should be good to go.

[robertlabrie]

I didn't have that event in the logs, but when I limited my results to 2015 the time parsed correctly. Set MAX_DAYS_AGO to 10000 and now it works fine. Thanks for your help. Wish splunk would just not index the data instead of consuming it wrong....

[woodcock]

I should not have mentioned the log because that would only have been generated if Splunk had read the timedate correctly and then ignored the log. In your case (I am not sure what causes the difference), Splunk instead assumes that the date (being so old?) is not in the proper format/place so it just uses some other time (previous event or current time) to date the event. I see things happen both ways with very old dates and it would be nice if I could figure out why sometimes it handles it one way vs. the other.