Getting Data In

How do I edit my props.conf for proper timestamp extraction?

hvaithia
Path Finder

My log sample event looks like this
"id": "2015-03-02_20-10-12", "keepLog": false
"id": "2015-03-19_10-26-38", "keepLog": false

etc. I have to extract this timestamp for the various events. My props.conf looks like below,

[json_no_timestamp]
TIME_PREFIX = :\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

I also tried

[json_no_timestamp]
TIME_PREFIX = "id":\s
TIME_FORMAT = %Y-%m-%d_%H-%M-%S

but still doesn't work. Its still showing the system time for the events. How should my TIME prefix and format look like?

0 Karma
1 Solution

hvaithia
Path Finder

Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N

TIMESTAMP_FIELDS = timestamp

TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true

TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)

View solution in original post

0 Karma

hvaithia
Path Finder

Thanks to Stephen and esix for their help on this. Really appreciate it. This is what worked for me finally
[json_timestamp]
INDEXED_EXTRACTIONS = json
TIME_FORMAT = %s%3N

TIMESTAMP_FIELDS = timestamp

TIME_PREFIX = timestamp
SHOULD_LINEMERGE = false
description = JavaScript Object Notation
category = Structured
disabled = false
pulldown_type = true

TIMESTAMP_FIELDS didn't unfortunately work with the forwarder (although it worked in the data preview)

0 Karma

piebob
Splunk Employee
Splunk Employee

when people provide useful information, please upvote them! i accepted your answer, since you provided what worked. 🙂

0 Karma

hvaithia
Path Finder

I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

To build on this...

   [json_no_timestamp]
  TIME_PREFIX = ^\"id\":\s\"
  TIME_FORMAT = %Y-%m-%d_%H-%M-%S

Look from the beginning of the line and also, just to be safe, escape the quotes..

hvaithia
Path Finder

[json_no_timestamp]
TIME_PREFIX=^\"id\":\s\"
TIME_FORMAT=%Y-%m-%d_%H-%M-%S
MAX_TIMESTAMP_LOOKAHEAD = 1048

Here is my updated props. I tried 512 and 1048 for timestamp look ahead, but its still logging the time when I create the log with my script. It's still not taking the time from the ID parameter.

Setting the lookahead will make splunk to look that many number of characters for the timestamp in every line. Did I get it right? I was wondering if I could match the 'timestamp' and deduce time with the epoch time if possible instead of getting it from 'id' field. Can you tell me how I can convert my epoch timestamp into something recognizable by splunk.

0 Karma

hvaithia
Path Finder

Also tried
[json_no_timestamp]
TIME_PREFIX="timestamp":
TIME_FORMAT=%s%3N
MAX_TIMESTAMP_LOOKAHEAD = 512

still nothing!!

0 Karma

hvaithia
Path Finder

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Try this

MAX_TIMESTAMP_LOOKAHEAD = 512
TIME_PREFIX  = \"id\"\: \"

This should capture the "id": " group, assuming this is the only id string in the event. You also should look at adjusting the lookahead depending on how far into the event the timestamp is..

0 Karma

stephanefotso
Motivator

Hello!
Try this:

  [json_no_timestamp]
 TIME_PREFIX = "id":\s\"
 TIME_FORMAT = %Y-%m-%d_%H-%M-%S
SGF

hvaithia
Path Finder

Thanks for the quick response. I think its my bad to have described my logs in part. But my actual logs look like below

{"building": false, "changeSet": {"items": [], "kind": null}, "builtOn": "rhel6", "description": null, "artifacts": [], "timestamp": 1430241584496, "number": 13, "actions": [{"causes": [{"upstreamBuild": 14, "shortDescription": "Started by upstream project \"answers\" build number 14", "upstreamProject": "answers", "upstreamUrl": "job/answers/"}]}, {}, {}, {}, {}, {"highlightsData": "[{\"Previous Job\":\"answers#14\"},{\"Previous Job\":\"answers_se\"},{\"Build host\":\"rhel6\"}]", "highlightsTable": "

Global Patterns

Previous Job:Started by answersPrevious Job:Started by answers #8Build host:Built on rhel6"}], "id": "2015-04-28_17-19-44", "keepLog": false, "url": "http://thefactory.xyz.com:9999/jenkins/job/answers/13/", "culprits": [], "result": "SUCCESS", "executor": null, "duration": 377658, "fullDisplayName": "answers", "estimatedDuration": 298415}

This is just one line. The entire splunk-logged file is a json dump. now I need to extract the time from either the 'ID' field or the 'timestamp' field. Will the same props work for these? (I tried it didn't). My ultimate aim is to have the logs logged in that timestamp (2015-04-28 in this case) instead of when I create this logs. Any help will be largely useful....

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...