Timestamp Extraction Issue

[himynamesdave]

My data looks like this:

{ EC_reference="C0000001", Entity_name="Charter 88", Entity_type="Third Party", Regulated_donee_type="", Recd_by="", Reported_under_62:12="", Is_sponsorship="", Donor_name="Joseph Rowntree Reform Trust, The Garden House", Donor_type="Company", Company_reg_num=":357963", Postcode="YO30 6WQ", Type_of_donation="Cash", Nature_Provision="", Purpose="", How_dealt_with="", Value_GBP="50000", Received_date="23-03-2001 00:00:00", Accepted_date="23-03-2001 00:00:00", Reported_date="07-09-2001 00:00:00", Compliance_breach="None" }

In my props.conf I have:

# your settings

NO_BINARY_CHECK=1

SHOULD_LINEMERGE=false

TIME_FORMAT=%d-%m-%Y %H:%M:%S

TIME_PREFIX=Accepted_date=\"

Though Splunk cannot ID a timestamp. I'm not sure what I've done wrong here...

[davebrooking · Accepted solution]

I think the issue could have something to do with the age of the timestamp, there is another setting MAX_DAYS_AGO that defaults to 2000 days (approx 5.5 years) your date is 2001 which is over 12 years ago. Try setting MAX_DAYS_AGO to its maximum value 10951 (approx 30 years)

[lukejadamec]

Nice catch

[himynamesdave]

BINGO! Thanks for your help.

[agodoy]

Not sure which field has the timestamp that you want to use. I have read of something like this working before when the time format alone would not.

TIME_FORMAT=Received_date="%d-%m-%Y %H:%M:%S"

[himynamesdave]

I was not aware you could use a text string in the TIME_FORMAT field. Looking at the docs it calls for a .

HOWEVER, when I change the props.conf file to the following, Splunk can identify the %H:%M:%S part of the timestamp :S

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=Accepted_date="%d-%m-%Y %H:%M:%S"

[lukejadamec]

The default MAX_TIMESTAMP_LOOKAHEAD is 150 characters. That might be why it's not finding it.

The setting for the log you posted would be 394 characters, but it looks like the size of the log can vary considerably.

Try setting MAX_TIMESTAMP_LOOKAHEAD = 1000

The documentation says this will effect indexing performance because it takes Splunk longer to find the timestamp:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Tunetimestampextractionforbetterindexingperf...

See this post for a similar example:

http://answers.splunk.com/answers/4338/time-stamping-problem-using-time_prefix

[himynamesdave]

Ah, OK. Although Splunk still fails to read the timestamp after removing the \

[lukejadamec]

You don't need to escape the = in the TIME_PREFIX

[himynamesdave]

Thanks for the suggestion, but declaring MAX_TIMESTAMP_LOOKAHEAD=x hasn't worker either 😞

# your settings
MAX_TIMESTAMP_LOOKAHEAD=1000
NO_BINARY_CHECK=1
SHOULD_LINEMERGE=false
TIME_FORMAT=%d-%m-%Y %H:%M:%S
TIME_PREFIX=Accepted_date\="