Hi,
I have a sample json file where I have to index the time. I created the stanza at props.conf.
{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009209525E8C08","volumeName":"VC-g0003v0011-H2","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
{"readIOps":0.0,"writeIOps":0.0,"otherIOps":0.0,"combinedIOps":0.0,"readThroughput":0.0,"writeThroughput":0.0,"combinedThroughput":0.0,"readResponseTime":0.0,"writeResponseTime":0.0,"combinedResponseTime":0.0,"volumeId":"0200000060080E500029094000009494525EB56B","volumeName":"dp0012v0044","readPhysicalIOps":0.0,"writePhysicalIOps":0.0,"observedTime":"2015-07-25T22:07:34.000+0000"}
[santa-banta]
INDEXED_EXTRACTIONS = json
TIME_PREFIX = "observedTime\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N+%4N
MAX_TIMESTAMP_LOOKAHEAD =
LINE_BREAKER =
SHOULD_LINEMERGE = false
TRUNCATE = 0
Questions:
Highly appreciate if anyone guide on this.
Following should work for you,
1) LINE_BREAKER is not required since you have SHOULD_LINEMERGE turned off, which means each line will be a new event.
2) MAX_TIMESTAMP_LOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIME_PREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED_EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.
PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAK_ONLY_BEFORE, BREAK_ONLY_AFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf
INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
Following should work for you,
1) LINE_BREAKER is not required since you have SHOULD_LINEMERGE turned off, which means each line will be a new event.
2) MAX_TIMESTAMP_LOOKAHEAD will look for number of characters defined to identify timestamp(in your case it is 28 characters after the TIME_PREFIX.
3) The same props.conf for your sourcetype should work for you even without INDEXED_EXTRACTIONS=json, only difference would be that automatic field extractions will not occur for JSON fields.
PS: There are several other props.conf properties which can be set for Timestamp and event extractions. like BREAK_ONLY_BEFORE, BREAK_ONLY_AFTER etc. Refer to the following documentation: http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf
INDEXED_EXTRACTIONS =json
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%Y-%m-%dT%H:%M:%S.%3N+%4N
TIME_PREFIX="observedTime":"
MAX_TIMESTAMP_LOOKAHEAD=28
Hi ,
The TIME_FORMAT you provided doesn't work for me.
My timestamp in data looks like: 2020-07-02T18:00:18+02:00 with name log_modified_date.
i have written below props.conf:
[_json]
INDEXED_EXTRACTIONS = json
KV_MODE = none
AUTO_KV_JSON = false
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = last_modified_date
TIME_FORMAT = %Y-%m-%dT%H:%M:%S+%2N:%2N
MAX_TIMESTAMP_LOOKAHEAD = 25
and getting time extracted as :
7/2/20
6:00:18.020 PM
I want the time field extracted in same way as in data with + value as well like:
7/2/20
6:00:18+02:00 PM something like this
Please let me know what i am doing wrong as i am not getting expected output.