I am having a difficult time extracting the correct timestamp from a specific log.
As you can see below, the beginning of the log entry there are two timestamps back to back.
2851,10/06/2011,18:59:29,10/06/2011,14:59:29,1011,
Here is my props.conf
[sourcetype]
MAX_TIMESTAMP_LOOKAHEAD = 20
TZ = America/New_York
TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}\:\d{2}\:\d{2},
The first timestamp is still being extracted. See anything wrong here?
EDIT: Can I perform these actions on a Universal Forwarder? Now that I think about it, I can't. Only a Heavy forwarder, correct?
EDIT: Looks like this was my fault. I had been trying this on a Universal Forwarder - I moved this to my Heavy Forwarder and it works! Thanks.
Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities
Yep, only on an install that supports indexing (which light and universal disable). http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Forwardercapabilities
Thanks! I will move the props.conf to our Heavy Forwarder.
I guess I was a little unclear on which comma. I meant the one in the middle of the date and time, not the one on the end.
10/06/2011*,*14:59:29,
The extra character in MAX_TIMESTAMP_LOOKAHEAD shouldn't stop this from working. In fact, sometimes to be safe I'll intentionally add an extra digit if there things like a space or comma. I haven't noticed any strange behavior because of this, but the approach mentioned here is a valid.
I've never really used the TIME_PREFIX, however from what I'm reading, the parser will try and find the TIME_PREFIX pattern first (which should match 2851,10/06/2011,18:59:29,) and then look the next 20 characters (from MAX_TIMESTAMP_LOOKAHEAD), which should then match 10/06/2011,14:59:29, which includes the comma. And since there is a comma, its not a "true" timestamp, as you will only have either a date or a time in the string. So it is never actually finding the second timestamp. There is also no need to escape the colons.
Try this:
TIME_FORMAT = %m/%d/%Y,%H:%M:%S
TIME_PREFIX = \d{4},\d{2}/\d{2}/\d{4},\d{2}:\d{2}:\d{2},
More details here
Your TIME_PREFIX looks okay in my estimation. It also looks like your MAX_TIMESTAMP_LOOKAHEAD is correct. If that isn't working, I would suggest you try specifying TIME_FORMAT as well. This needs to be done where the data is parsed, so on the Heavy Forwarder, or on the indexer which the forwarder is sending, presuming a forwarder is part of the picture.
http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition
I think that this should be the correct format:
TIME_FORMAT = %m/%d/%Y,%H:%M:%S
corrected. thanks!
Your %Y needs capitalized to match the 4 digit year.
As far as I can see your MAX_TIMESTAMP_LOOKAHEAD = 20 only gets you as far as "9:29,10/06/2011,14:59:29,1011,"
Try changing your MAX_TIMESTAMP_LOOKAHEAD=25.
Thanks for the answer, I tried this and no luck!
This is actually not correct. Using TIME_PREFIX tells splunk what precedes the timestamp. When MAX_TIMESTAMP_LOOKAHEAD is used in conjunction with TIME_PREFIX, you are telling splunk to look ahead as many characters as you've specified after matching the regex in TIME_PREFIX. In this case, it would start at the second timestamp and look 20 characters ahead, means it should see '10/06/2011,14:59:29' as the timestamp.
Oops i got confused on this! Indeed your MAX_TIMESTAMP_LOOKAHEAD is correct.