Getting Data In

How to avoid wrong timestamp with MAX_TIMESTAMP_LOOKAHEAD

ctaf
Contributor

Hello,

I have logs from Cisco ESA (emails) and some of them are logged in the futur. For example this log is marked "12/16/15 11:20:30.290 PM" by Splunk, but it should be "Dec 16 00:28:26".

Dec 16 00:28:26 <internal-IP> logs: Info: MID XXXXX Message-ID '<20151215232030290.BLABLA@XXXX.COM>'

As you can see, the email address contains "20:30.290" which makes the timestamp wrong.
To block this behavior, I tried to setup MAX_TIMESTAMP_LOOKAHEAD to 16 or 19, but it seems that Splunk keeps reading the whole log because for the above example, I have this field: timeendpos = 82 (it stops after 20151215232030290)

My props.conf (on the indexer:)

[cisco:esa:legacy]
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 19
NO_BINARY_CHECK=true

Any idea ?

Thank you

1 Solution

javiergn
SplunkTrust
SplunkTrust

You can try this too:

[cisco:esa:legacy]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19
NO_BINARY_CHECK=true
TIME_FORMAT = %b %d %H:%M:%S

View solution in original post

egray1812
Engager

From Settings > Add Data
You can upload any file to Splunk to analyze and apply pre-trained sourcetypes (or create your own) with your specifications.

When you're finished with "Set Source Type", at the bottom go to "Advanced" > "Copy to Clipboard" - this will produce the settings.

hope this saves some time?

0 Karma

hettervik
Builder

Hi,

Did you find a solution to the MAX_TIMESTAMP_LOOKAHEAD problem? I have the same issue as you had. I've noticed the answers you've written below, that you have renamed the sourcetype, but I don't quite get it. Is there some other fields that somehow overrides the MAX_TIMESTAMP_LOOKAHEAD field?

Here are some of the fields configured on my sourcetype. Do these seem okay to you?

SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = ^\d{2,}:\d{2,}:\d{2,},\d{3,}
MAX_TIMESTAMP_LOOKAHEAD = 20
0 Karma

hettervik
Builder

I found out why my MAX_TIMESTAMP_LOOKAHEAD wasn't working. A heavy forwarder was parsing the data, and thus marking it as "cooked" before it reached my indexer. When cooked data gets to an indexer it isn't re-parsed, and all source type settings on the indexer aren't taken into account. Case closed.

0 Karma

burwell
SplunkTrust
SplunkTrust

Wow! I spent hours tonight debugging this and you found me the solution hettervi. Thanks so much!

masonmorales
Influencer

So, as others have mentioned, the first timestamp in the event does not contain a year, so TIME_FORMAT and TIME_PREFIX are being ignored. I was actually able to parse the timestamp (20151215232030290) in the event using Splunk's default settings for timestamp detection. Can you try something like:

[cisco:esa:legacy]
SHOULD_LINEMERGE = false
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
category = Custom
disabled = false
MAX_TIMESTAMP_LOOKAHEAD = 128
pulldown_type = true

You can specify TZ = (time zone of the data) as well. You may need to restart Splunk for the change to take effect.

0 Karma

ctaf
Contributor

I think I found the problem. I had a sourcetype renaming (from "cisco:esa" to "cisco:esa:legacy"). And the timestamp configuration were on the cisco:esa:legacy sourcetype. I just read that renaming works only for search time. I thought it was index-time.

Thank you for your time.

0 Karma

javiergn
SplunkTrust
SplunkTrust

You can try this too:

[cisco:esa:legacy]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19
NO_BINARY_CHECK=true
TIME_FORMAT = %b %d %H:%M:%S

dieguiariel
Path Finder

Just to thank for this, for several weeks i've been trying to pinpoint a trouble with one esa (we have two, the logs from one were indexed correctly but we couldn't find the logs from the other).

This thread made me realize that that the ip from the trouble one was XXX.XXX.X20.19 so splunk was indexing the events on 2019.

Side effect was the number of small buckets, first we thougth there were separate problems but it was the same reason. With this solution the events are indexed correctly and the alert for the number of small buckets desapeared.

From other thread (about small buckets:  "The main cause of this issue is most likely going to be because the timestamps on the data you are feeding in are all over the place. Splunk wants to be mostly chronological, so the buckets contain data from a certain window of time." )

Thanks!

Tags (1)
0 Karma

javiergn
SplunkTrust
SplunkTrust

Run btool against your props file in order to make sure the right properties are being applied.

Maybe your time_prefix is causing some problems. Use something like this instead:

[cisco:esa:legacy]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 19
NO_BINARY_CHECK=true
TIME_FORMAT = %b %d %H:%M:%S
TIME_PREFIX = .+

Also take a look at the following in order to make sure your time_prefix and lookahead are behaving as expected: http://docs.splunk.com/Documentation/Splunk/6.3.2/Data/Configuretimestamprecognition

0 Karma

ctaf
Contributor

I think I found the problem. I had a sourcetype renaming (from "cisco:esa" to "cisco:esa:legacy"). And the timestamp configuration were on the cisco:esa:legacy sourcetype. I just read that renaming works only for search time. I thought it was index-time.

Thank you for your time.

0 Karma

ctaf
Contributor

I will try it. But it is weird that MAX_TIMESTAMP_LOOKAHEAD is not taken in account...

0 Karma

javiergn
SplunkTrust
SplunkTrust

I think MAX_TIMESTAMP_LOOKAHEAD is being ignored because "Dec 16 00:28:26" does not contain a valid year, whereas 20151215232030290 does.

ctaf
Contributor

OK I tried it, but it doesn't work 😞
I still have timestamp issues...

0 Karma

jkat54
SplunkTrust
SplunkTrust

Your TIME_PREFIX is set to ^ which is the beginning of the line in regex.

Try this instead

TIME_PREFIX = "< "
0 Karma

ctaf
Contributor

This is not what I want. "Dec 16 00:28:26" is the true date

0 Karma

jkat54
SplunkTrust
SplunkTrust

I just thought Splunk would auto recognize the time stamp if you correctly identified it's location on the line.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...