Splunk Search

Why can't I capture the correct timestamp for events with my current regex?

shariinPH
Contributor

Hi guys,

I'm having trouble in getting the right timestamp from my log file.
Please refer to this image .. http://postimg.org/image/mvzxzyfzv/
Splunk is getting 4/16/2015 as the _time, but what i want is to get the first field in the example which is 04/20/2015 HH:MM:SS
I already tried using some regex like ^\"\d+\/\d+\/\d_\s\d+\:\d+\:\d+ to get the timestamp, but it's failing to get the right time.

Please help me guys!
Thanks!

Tags (2)
0 Karma
1 Solution

schose
Builder

Hi,

Your issue is that your events are more than 2 days in the future.. splunk will ignore timestamps for more than 2 days in future.

You can use the MAX_DAYS_HENCE Parameter in props.conf go get it working.

Regards,

Andreas

View solution in original post

stephanefotso
Motivator

Try this:

........|rex "\"(?<date>\d+\/\d+\/\d+\s\d+\:\d+\:\d+)\. "|table date
SGF
0 Karma

shariinPH
Contributor

@stephanefotso, this may work also, but I need to put the regex on the props.conf so that before indexing the data,it will capture the field that i want to be the time stamp of my event.

0 Karma

schose
Builder

Hi,

Your issue is that your events are more than 2 days in the future.. splunk will ignore timestamps for more than 2 days in future.

You can use the MAX_DAYS_HENCE Parameter in props.conf go get it working.

Regards,

Andreas

shariinPH
Contributor

hello @schose, this may work but the scenario is i have a log file consists of data from Mar 11 2015 up until today. So i need to index all of those. from my previous config in props.conf I successfully capture the right timestamp from dates Mar 11 to April 16, when splunk see's the log from Mar 17, it gets the timestamp from another field, not on the field that I needed.

0 Karma

shariinPH
Contributor

@schose maybe i can use MAX_TIMESTAMP_LOOKAHEAD right? and set it to 12 or 41? what do you think ?

0 Karma

schose
Builder

Hey your screenshot it showing a date from 20th of april... can you use a older logfile, reproduce and take a screenshot of the issue - cause I'm sure it's the issue - i tested it out.

Regards,

0 Karma

shariinPH
Contributor

hi @schose here's the screen shots:
http://postimg.org/image/n57egmbmb/
http://postimg.org/image/b2qp8zbif/
its just a sample, i've got a lot of those events that didnt get my desired timestamp
my desired timestamp is the one enclosed in the green rectangle

my props.conf

[download_logs_ftp_two]
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIME_PREFIX = ^\d+\,\"
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true
0 Karma

jeffland
SplunkTrust
SplunkTrust

No. If splunk is told that a timestamp can be no further in the future than two days, it will not accept a timestamp from further in the future even if this is the only timestamp there is in an event. As schose said, you need to set MAX_DAYS_HENCE to a higher value; it shouldn't harm your system to incease it a few days.

0 Karma

shariinPH
Contributor

hello @jeffaland wat if i use MAX_TIMESTAMP_LOOKAHEAD ? i cannot use the MAX_DAYS_HENCE right now because i have logs from MAr 2015. Am I right?

0 Karma

jeffland
SplunkTrust
SplunkTrust

Well there's two things you need to do:
a) you need to show splunk precisely which timestamp to use if there is more than one (such as in your screenshot) and
b) if (from todays perspective) your timestamp is supposed to be the 21st of April (also like in your screenshot), you need to allow those future timestamps.

0 Karma

shariinPH
Contributor

from my props.conf, i already put TIME_PREFIX so that splunk would capture the timestamp i needed. It works on majority of the events, but i have a problem with some of the events because it's not getting the timestamp i defined with the TIME_PREFIX

[download_logs_ftp_two]
 INDEXED_EXTRACTIONS = csv
 KV_MODE = none
 NO_BINARY_CHECK = true
 SHOULD_LINEMERGE = false
 TIME_PREFIX = ^\d+\,\"
 category = Structured
 description = Comma-separated value format. Set header and other settings in "Delimited Settings"
 disabled = false
 pulldown_type = true
0 Karma

shariinPH
Contributor

that's why I'm thinking of using MAX_TIMESTAMP_LOOKAHEAD so that splunk will look for timestamps from the defined integer, for like example default value is 150 so it looks like MAX_TIMESTAMP_LOOKAHEAD=150 so it would get characters in my event from 1-150. Is it right?

0 Karma

jeffland
SplunkTrust
SplunkTrust

Generally speaking, yes.

0 Karma

shariinPH
Contributor

what about the second thing i need to do? please elaborate ..

0 Karma

jeffland
SplunkTrust
SplunkTrust

The green rectangle you mentioned which represents your desired timestamp, it shows a date which is in the future. By default, splunk will only accept timestamps which are up to two days in the future; yours is three. So, as schose said, you'll need to set MAX_DAYS_HENCE to a higher value. This is done in props.conf, see here.

0 Karma

shariinPH
Contributor

so all in all i have to put in my props.conf the ff:

MAX_DAYS_HENCE=4
MAX_TIMESTAMP_LOOKAHEAD=29
TIME_PREFIX=^\d+\,\"

is it right?

0 Karma

shariinPH
Contributor

It works!!! Thanks guys! @schose @jeffland

Cheers! 🙂
i adjusted the MAX_DAYS_HENCE from 4 to 15 just in case some logs will came in in advance.

jeffland
SplunkTrust
SplunkTrust

You should try it and be able to see if it works 🙂 but that's what I think is needed.
If some of your dates are even further in the future, setting MAX_DAYS_HENCE to the required amount might be needed as well.

shariinPH
Contributor

Thanks! will try this 🙂 hope it works .. 🙂

0 Karma

shariinPH
Contributor

@jeffland in my logs, you can see lots of dates and time, so i have to specify the field where splunk will caputure my defined timestamp. as of now splunk is getting the time and date somewhere in my logs which is not my defined pattern or not my desired timestamp to use.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...