Another timestamp question

[Greg_LeBlanc]

I am having a difficult time extracting the correct timestamp from a specific log.

As you can see below, the beginning of the log entry there are two timestamps back to back.

2851,10/06/2011,18:59:29,10/06/2011,14:59:29,1011, 

Here is my props.conf

    [sourcetype]
    MAX_TIMESTAMP_LOOKAHEAD = 20
    TZ = America/New_York
    TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}\:\d{2}\:\d{2},

The first timestamp is still being extracted. See anything wrong here?

EDIT: Can I perform these actions on a Universal Forwarder? Now that I think about it, I can't. Only a Heavy forwarder, correct?

EDIT: Looks like this was my fault. I had been trying this on a Universal Forwarder - I moved this to my Heavy Forwarder and it works! Thanks.

[alacercogitatus · Accepted solution]

Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities

[Greg_LeBlanc]

Thanks! I will move the props.conf to our Heavy Forwarder.

[alacercogitatus]

I guess I was a little unclear on which comma. I meant the one in the middle of the date and time, not the one on the end.

10/06/2011*,*14:59:29,

[jbsplunk]

The extra character in MAX_TIMESTAMP_LOOKAHEAD shouldn't stop this from working. In fact, sometimes to be safe I'll intentionally add an extra digit if there things like a space or comma. I haven't noticed any strange behavior because of this, but the approach mentioned here is a valid.

[alacercogitatus]

I've never really used the TIME_PREFIX, however from what I'm reading, the parser will try and find the TIME_PREFIX pattern first (which should match 2851,10/06/2011,18:59:29,) and then look the next 20 characters (from MAX_TIMESTAMP_LOOKAHEAD), which should then match 10/06/2011,14:59:29, which includes the comma. And since there is a comma, its not a "true" timestamp, as you will only have either a date or a time in the string. So it is never actually finding the second timestamp. There is also no need to escape the colons.

Try this:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S

TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}:\d{2}:\d{2},

More details here

[jbsplunk]

Your TIME_PREFIX looks okay in my estimation. It also looks like your MAX_TIMESTAMP_LOOKAHEAD is correct. If that isn't working, I would suggest you try specifying TIME_FORMAT as well. This needs to be done where the data is parsed, so on the Heavy Forwarder, or on the indexer which the forwarder is sending, presuming a forwarder is part of the picture.

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

I think that this should be the correct format:

TIME_FORMAT = %m/%d/%Y,%H:%M:%S

[jbsplunk]

corrected. thanks!

[alacercogitatus]

Your %Y needs capitalized to match the 4 digit year.

[jdunlea_splunk]

As far as I can see your MAX_TIMESTAMP_LOOKAHEAD = 20 only gets you as far as "9:29,10/06/2011,14:59:29,1011,"

Try changing your MAX_TIMESTAMP_LOOKAHEAD=25.

[Greg_LeBlanc]

Thanks for the answer, I tried this and no luck!

[jbsplunk]

This is actually not correct. Using TIME_PREFIX tells splunk what precedes the timestamp. When MAX_TIMESTAMP_LOOKAHEAD is used in conjunction with TIME_PREFIX, you are telling splunk to look ahead as many characters as you've specified after matching the regex in TIME_PREFIX. In this case, it would start at the second timestamp and look 20 characters ahead, means it should see '10/06/2011,14:59:29' as the timestamp.

[jdunlea_splunk]

Oops i got confused on this! Indeed your MAX_TIMESTAMP_LOOKAHEAD is correct.