Splunk Search

Why is my TIME_FORMAT regular expression in props.conf not working for an index time extraction?

sreejith2k2
Explorer

HI I am using following regular expression for the index time extraction in the props.conf. For some reason, it is not extracting properly.

Event: 2017-03-15T11:30:02.609835+00:00 postfix/pickup[19819]: 89389386: uid=0 from user1

I have defined my sourcetype as mail.

[source::...mail]
sourcetype=mail

[mail]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6Q%:z
TIME_PREFIX = ^
LINE_BREAKER = ([\r\n]+)
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE=5000

Also, is there any websites i can test the TIME_FORMAT regular expression similar to regex101.com?

0 Karma
1 Solution

jwelch_splunk
Splunk Employee
Splunk Employee

What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF

Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32

And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}

View solution in original post

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

What is the source of the data? is it /var/log/mail/*.log and how is it coming to the indexer? Is there a UF

Other things I noticed:
increase MAX_TIMESTAMP_LOOKAHEAD = 32

And it is always best to do your LINE_BREAKER like so
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}+\d{2}:\d{2}

0 Karma

sreejith2k2
Explorer

HI Welch,

Thanks for the answer. There were 2 issues

  1. MAX_TIMESTAMP_LOOKAHEAD
  2. [source::...mail] - i have put only 2 dots in my props.conf instead of 3.
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The TIME_FORMAT attribute does not use regex strings.
I'm not aware of any website for testing time format strings, but you can do it in a search window. Try something like this:

| makeresults | eval TS="2017-03-15T11:30:02.609835+00:00" | eval epoch=strptime(TS,"%Y-%m-%dT%H:%M:%S.%6Q%:z") | table TS epoch

If the format string is bad, epoch will be null.

Your format string works for me in search, but I haven't tried it at index time. You might want to try this alternative: "%Y-%m-%dT%H:%M:%S.%6N%z".

---
If this reply helps you, Karma would be appreciated.

sreejith2k2
Explorer

Thanks Rich for your search. This search helped me in reducing the time in doing the testing..

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...