The raw data looks like:
... blah, blah, blah ... "detail-type": "GuardDuty Finding", "time": "2019-03-14T14:40:39Z"}
On our Heavy Forwarder I've setup in Splunk_TA_aws/local
[aws:kinesis]
TIME_PREFIX = \s+\"time\":\s+\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
MAX_TIMESTAMP_LOOKAHEAD = 40960
The regex works on the standalone Splunk instance I have on my laptop, it works on regex101.com
But when the data is indexed, the time Splunk indexes on my cluster is: 3/14/19 2:46:29.090 PM
Any clues as to what might be happening here?
Support came up with the fix for this issue. Setting use_hec = false under [global_settings] in aws_kinesis.conf resolved my problem, and the time being stamped now matches the time in the json.
en-US may be? (locale)
Thank you ...
LANG is en_US.UTF-8, CHARSET (in btool) is also UTF-8
Same goes for my laptop.
Cheers - PK
If you switch your locale to en_GB or another locale which uses the DAY/MONTH/YEAR format, it will reformat the timestamps for you.
Thank you. I'm not looking to have the time reformatted. Only to match what's in the event, yet no matter what I do, Splunk seems to set _time to the time that the event was indexed.
detail-type time _time index_time
GuardDuty Finding 2019-03-15T16:30:02Z 2019-03-15 16:31:03.682 03/15/2019 16:31:04
GuardDuty Finding 2019-03-15T16:30:06Z 2019-03-15 16:31:01.686 03/15/2019 16:31:02
GuardDuty Finding 2019-03-15T16:25:11Z 2019-03-15 16:26:13.278 03/15/2019 16:26:13
I understand now.
Your max time stamp lookahead is wrong.
Set it to 20
The lookahead begins after the prefix match, not the start of the event, so smaller is better and faster, and could be your issue.
Thanks again,
Yes, I've played around with multiple values for lookahead and still end up with about a 1 minute difference between the value of 'time" and what Splunk stamps ... I've opened up a ticket with support. It appears that the collection is doing its' own thing. ( Not the first issue we've had with the AWS Add-On. )
Cheers.