Getting Data In

Another timestamp question

Greg_LeBlanc
Path Finder

I am having a difficult time extracting the correct timestamp from a specific log.

As you can see below, the beginning of the log entry there are two timestamps back to back.

2851,10/06/2011,18:59:29,10/06/2011,14:59:29,1011, 

Here is my props.conf

    [sourcetype]
    MAX_TIMESTAMP_LOOKAHEAD = 20
    TZ = America/New_York
    TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}\:\d{2}\:\d{2},

The first timestamp is still being extracted. See anything wrong here?

EDIT: Can I perform these actions on a Universal Forwarder? Now that I think about it, I can't. Only a Heavy forwarder, correct?

EDIT: Looks like this was my fault. I had been trying this on a Universal Forwarder - I moved this to my Heavy Forwarder and it works! Thanks.

1 Solution

alacercogitatus
SplunkTrust
SplunkTrust

Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities

View solution in original post

alacercogitatus
SplunkTrust
SplunkTrust

Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities

Greg_LeBlanc
Path Finder

Thanks! I will move the props.conf to our Heavy Forwarder.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

I guess I was a little unclear on which comma. I meant the one in the middle of the date and time, not the one on the end.

10/06/2011*,*14:59:29,

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

The extra character in MAX_TIMESTAMP_LOOKAHEAD shouldn't stop this from working. In fact, sometimes to be safe I'll intentionally add an extra digit if there things like a space or comma. I haven't noticed any strange behavior because of this, but the approach mentioned here is a valid.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

I've never really used the TIME_PREFIX, however from what I'm reading, the parser will try and find the TIME_PREFIX pattern first (which should match 2851,10/06/2011,18:59:29,) and then look the next 20 characters (from MAX_TIMESTAMP_LOOKAHEAD), which should then match 10/06/2011,14:59:29, which includes the comma. And since there is a comma, its not a "true" timestamp, as you will only have either a date or a time in the string. So it is never actually finding the second timestamp. There is also no need to escape the colons.

Try this:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S

TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}:\d{2}:\d{2},

More details here

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

Your TIME_PREFIX looks okay in my estimation. It also looks like your MAX_TIMESTAMP_LOOKAHEAD is correct. If that isn't working, I would suggest you try specifying TIME_FORMAT as well. This needs to be done where the data is parsed, so on the Heavy Forwarder, or on the indexer which the forwarder is sending, presuming a forwarder is part of the picture.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

I think that this should be the correct format:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S
0 Karma

jbsplunk
Splunk Employee
Splunk Employee

corrected. thanks!

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Your %Y needs capitalized to match the 4 digit year.

0 Karma

jdunlea_splunk
Splunk Employee
Splunk Employee

As far as I can see your MAX_TIMESTAMP_LOOKAHEAD = 20 only gets you as far as "9:29,10/06/2011,14:59:29,1011,"

Try changing your MAX_TIMESTAMP_LOOKAHEAD=25.

0 Karma

Greg_LeBlanc
Path Finder

Thanks for the answer, I tried this and no luck!

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

This is actually not correct. Using TIME_PREFIX tells splunk what precedes the timestamp. When MAX_TIMESTAMP_LOOKAHEAD is used in conjunction with TIME_PREFIX, you are telling splunk to look ahead as many characters as you've specified after matching the regex in TIME_PREFIX. In this case, it would start at the second timestamp and look 20 characters ahead, means it should see '10/06/2011,14:59:29' as the timestamp.

0 Karma

jdunlea_splunk
Splunk Employee
Splunk Employee

Oops i got confused on this! Indeed your MAX_TIMESTAMP_LOOKAHEAD is correct.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...