My log sample event looks like this
"id": "2015-03-02_20-10-12", "keepLog": false
"id": "2015-03-19_10-26-38", "keepLog": false
etc. I have to extract this timestamp for the various events. My props.conf looks like below,
[json_no_timestamp]
TIME_PREFIX = :\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S
I also tried
[json_no_timestamp]
TIME_PREFIX = "id":\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S
but still doesn't work. Its still showing the system time for the events. How should my TIME prefix and format look like?
Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)
Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N
TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true
TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)
when people provide useful information, please upvote them! i accepted your answer, since you provided what worked. 🙂
I think its my bad to have described my logs in part. But my actual logs look like below
{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "
This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....
To build on this...
[json_no_timestamp]
TIME_PREFIX = ^\"id\":\s\"
TIME_FORMAT = %Y-%m-%d_%H-%M-%S
Look from the beginning of the line and also, just to be safe, escape the quotes..
[json_no_timestamp]
TIME_PREFIX=^\"id\":\s\"
TIME_FORMAT=%Y-%m-%d_%H-%M-%S
MAX_TIMESTAMP_LOOKAHEAD = 1048
Here is my updated props. I tried 512 and 1048 for timestamp look ahead, but its still logging the time when I create the log with my script. It's still not taking the time from the ID parameter.
Setting the lookahead will make splunk to look that many number of characters for the timestamp in every line. Did I get it right? I was wondering if I could match the 'timestamp' and deduce time with the epoch time if possible instead of getting it from 'id' field. Can you tell me how I can convert my epoch timestamp into something recognizable by splunk.
Also tried
[json_no_timestamp]
TIME_PREFIX="timestamp":
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD = 512
still nothing!!
Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below
{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "
This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....
Try this
MAX_TIMESTAMP_LOOKAHEAD = 512
TIME_PREFIX = \"id\"\: \"
This should capture the "id": " group, assuming this is the only id string in the event. You also should look at adjusting the lookahead depending on how far into the event the timestamp is..
Hello!
Try this:
[json_no_timestamp]
TIME_PREFIX = "id":\s\"
TIME_FORMAT = %Y-%m-%d_%H-%M-%S
Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below
{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "
This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....