DateParserVerbose - Failed to parse timestamp Warning

[srubik]

We have a log file in our environment which writes a timestamp followed by a lot of data on new lines. The number of data lines is variable and each line needs a slightly different regular expression. For example:

09/06/2013 10:00:00 This is a new event
This is a bunch of data
This is more data
09/06/2013 10:01:00 This is a new event 2
This is a bunch of data
This is more data
This is even more data
And even more data 

We currently have SHOULD_LINEMERGE set to false in props.conf so make our regular expression in transforms.conf easier to work with. If we set this to true, our regular expression would be huge and confusing having to deal with one really large event. So far, things are working great. As events come in, the data without the timestamp takes the timestamp of the previous event that had a timestamp which is exactly what we want. However, our splunkd.log is filling up with the following warning:

09-06-2013 08:50:00.897 -0500 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Fri Sep  6 08:50:00 2013).

Is there any way we can prevent this warning from occurring? Like I said, Splunk is doing the right thing and setting the events without a timestamp to the previous event timestamp. And we like having the short regular expressions and not having to worry about treating a bunch of lines as one huge event. Is there something we can set in props.conf that tell Splunk that this is indeed what we want to happen to quiet down the warning?

[yannK]

Hi Srubik, the behavior is totally expected.
Splunk considers each line as a separate events, and will warn you that some lines do no contains a valid timestamp (after an expensive and systematic scan).

There are only 3 solutions here :

• use a valid timestamp in each line
• use multiline events (SHOULD_LINEMERGE=true see http://docs.splunk.com/Documentation/Splunk/5.0.4/Data/Indexmulti-lineevents )
• or change the log level of the DateParserVerbose component to ERROR instead of WARN. (http://docs.splunk.com/Documentation/Splunk/5.0.4/Troubleshooting/Enabledebuglogging )

[triest]

You can also use LINE_BREAKER to have multi-line events without using SHOULD_LINEMERGE. We do that a lot since line merging isn't efficient. It's covered pretty well in the docs, but if some one needs help let me know.

[rturk]

I don't think I understand the problem with using SHOULD_LINEMERGE. It sounds more like (potentially) inefficient regex use. Given your example, best practice would be to configure these as two events, instead of 8 events, which would give you the correct timestamping, event breaking, and (what should be) easy regex-based extractions.

Is it possible that you edit your question with your props.conf, transforms.conf, and a couple of sample events (feel free to redact them so long as the regex still matches).