HI I am using following regular expression for the index time extraction in the props.conf. For some reason, it is not extracting properly.
Event: 2017-03-15T11:30:02.609835+00:00 postfix/pickup[19819]: 89389386: uid=0 from user1
I have defined my sourcetype as mail.
[source::...mail]
sourcetype=mail
[mail]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
TIME_PREFIX = ^
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE=5000
Also, is there any websites i can test the TIME_FORMAT regular expression similar to regex101.com?
What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF
Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32
And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}
What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF
Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32
And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}
HI Welch,
Thanks for the answer. There were 2 issues
The TIME_FORMAT attribute does not use regex strings.
I'm not aware of any website for testing time format strings, but you can do it in a search window. Try something like this:
| makeresults | eval TS="2017-03-15T11:30:02.609835+00:00" | eval epoch=strptime(TS,"%Y-%m-%dT%H:%M:%S.%6Q%:z") | table TS epoch
If the format string is bad, epoch will be null.
Your format string works for me in search, but I haven't tried it at index time. You might want to try this alternative: "%Y-%m-%dT%H:%M:%S.%6N%z"
.
Thanks Rich for your search. This search helped me in reducing the time in doing the testing..