Manually configure timestamp at index time from custom datetime.xml

[bizza]

I tried to configure a custom datetime.xml (for my first time) as follow:

<datetime>

<define name="csdate" extract="year, month, day, hour, minute">
        <text><![CDATA[[\s\S]{40}(\d{4})(\d{2})(\d{2})[\s\S]{206}(\d{2})(\d{2})]]></text> 
</define>

<timePatterns>
    <use name="csdate"/>
</timePatterns> 

<datePatterns>
    <use name="csdate"/>
</datePatterns>
</datetime>

Regex match exactly year, mont, day, hour and minute.
In props.conf I added

DATETIME_CONFIG = /etc/system/local/datetime.xml

SHOULD_LINEMERGE = FALSE

TIME_FORMAT = %Y%m%d%H%M

Any ideas why data are not indexed with resulting timestamp?
I tried to split in 2 regex, for timePatterns and datePatterns match, but the result is still the same.

Or do you suggest a different way to achieve timestamp override at index time?

Regards

[abonuccelli_spl · Accepted solution]

Problem here I believe was due to actual timestamps in raw event past the default (MAX_TIMESTAMP_LOOKAHEAD) 150 chars.

[bizza]

Yes, problem was MAX_TIMESTAMP_LOOKAHEAD.
Thanks for your help guys

ciao

[abonuccelli_spl]

MMM, well it works for me...
Bizza should be able to confirm
Antonio

[marcoscala]

Antonio,
I'm afraid but that's not the case either. In his/her case, date and time are splitted in the event data, so usual timeformat is more complex to manage. Unfortunately we can't use REGEX for TIME_FORMAT, otherwise that was the solution.

Marco

[lguinn2]

You don't need a custom datetime.xml - I wouldn't do it that way. It is complicated and unnecessary.

In props.conf all you should need is

SHOULD_LINEMERGE = FALSE
TIME_FORMAT = %Y%m%d%H%M

Assuming that your timestamp looks like

201405021209

If not, please comment with an example or two of the timestamp.

[bizza]

Yes, I added the --Y --m ecc only to show where timestamp fields are.
Ignore it and you'll have the original log line.

[nitesh218ss]

Hi in my log event and filename date is not present i want give a fix date to log so what is do ?

[lmyrefelt]

Did you add the --Y and --m into the event example as an clarification ?
otherwise you could try;
TIME_PREFIX = \d{numberOfDigits}\s++
MAX_TIMESTAMP_LOOKAHEAD = 20
TIME_FORMAT = %Y%m%d
TIME_FORMAT = --Y%Y--m%m--d%d

[bizza]

The problem is that timestamp is splitted on every lines.
For example:

204023600511105443000 20140422--Y2014--m04--d180000000005.0600000000000041096125031ABDCE 81234567 ABDCE F & C 10024 ABDCE F & C 45399700123456789000000000.104023600582105443000 386511186636492--H15--M36PSBP

every line has 300 characters(digits), fields are position-sensitive.
I added --Y, --m, --d, --H and --M just before timestamp fields.

I believe that a custom datetime.xml is my only option.