Splunk Dev

Unable to extract new timestamp field

wuming79
Path Finder

Hi,

I have a log file with timestamp = time of saving the file thus the timestamp from splunk are all the same. I not want to extract the epoch time example "1495447178314" and make it as my timestamp. Is that possible?

I also tried to extract new field using regular expression \d{13}[^\}\}]* but could not get anything when I applied it. Using https://regex101.com/ to test it out, somehow it looks like there were 2 matches. Can someone help? Maybe is it the way I do the \}\} ?

15199256 [EPS-log-dispatcher-9] INFO 
1.24978294676695149906 - {"Log Header": "{"endpointKeyHash":null,"applicationToken":null,"headerVersion":null,"timestamp":null,"logSchemaVersion":null}", "Event": {"temperature":35,"timeStamp":1495447178314}}
Tags (1)
0 Karma
1 Solution

maciep
Champion

I think you should be able to use TIME_PREFIX to tell Splunk where the timestamp is and %s to tell Splunk the format. These setting need to go in props.conf. If the input is coming from a universal forwarder, then the props config should be on your indexer likely.

TIME_PREFIX = "timeStamp":
TIME_FORMAT = %s

There was a similar answer here: https://answers.splunk.com/answers/111161/how-do-i-get-splunk-to-recognise-epoch-time.html

View solution in original post

0 Karma

wuming79
Path Finder

Hi,

I managed to make the time format from Epoch to human readable but I can't really get the millisecond out.

Example timeStamp":1495447178314
From Splunk it converted to "5/22/17 5:59:38.000 PM" but from https://www.epochconverter.com/, it is showing
May 22, 2017 5:59:38.314 PM

Reference document: http://docs.splunk.com/Documentation/Splunk/6.0/Data/Configuretimestamprecognition, .%3N should show the milliseconds.

alt text

alt text

Another weird thing is, I had to use %s%s.%3N to show the time in my time zone. If I use %s.%3N, it will show all my time to be "12/31/99 11:59:59.999 PM". Am I supposed to use just %s.%3N?

0 Karma

wuming79
Path Finder

I managed to extract my new field but it seems not what I wanted...the "Time" still 6/10/17.....

alt text

0 Karma

maciep
Champion

i'm having a little trouble reading your screenshots. But those settings should be in props.conf for the sourcetype of the log you're ingesting. I see you referenced the stanza for custom_log, but what about for the IoT Temperature sourcetype? The timestamp settings need to be applied there.

0 Karma

wuming79
Path Finder

Actually, I didn't know how and where to edit the props.conf ...
I think I used the UI to change the timestamp format and time prefix and saved a new sourcetype? Is this ok?

[http://imgur.com/4Nn9Bf1][1]

0 Karma

maciep
Champion

So are you just uploading the file in the GUI over an over? Not sure what your env looks like, how you plan to ingest this data, etc. It doesn't matter necessarily what you call the sourcetype, but that you are using the same sourcetype when you search for the data as when you ingest it.

So is the timestamp correct in Splunk yet? Maybe not to the ms, but is it no longer 6/10? I'm having trouble following along with all of these posts.

0 Karma

wuming79
Path Finder

The latest I got the the post below with EpochCOnverter screenshot. Actually, the above screenshots are all different. I think we can skip the above screenshots now since I managed to get the timestamp now but with the milliseconds still having some issues.

0 Karma

maciep
Champion

So the timestamp of the splunk event is correct (minus milliseconds)? You're not just extracting the timestamp to a new field, right? There is a big difference, so want to be sure splunk timestamping is working as expected.

0 Karma

wuming79
Path Finder

yup, timestamp working minus milliseconds and I'm no longer extracting it as a new field.

0 Karma

maciep
Champion

Ok. I'm not sure about the millseconds part, maybe %s%3N?

0 Karma

wuming79
Path Finder

oh....you were right...example has a dit and I just follow but my data did not have. Now it shows the milliseconds...thanks.

0 Karma

wuming79
Path Finder

Do I just add to 1st 2 rows of props.conf file below [custom_log]?

Currently my props.conf file is as follows:

[custom_log]
DATETIME_CONFIG = 
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
description = For IoT Demo
disabled = false
maxDist = 75
pulldown_type = true

I tried to upload the data again but it still shows the file saved time instead of the timestamp. Is there other ways such as extracting it as another field?
alt text

alt text

0 Karma

maciep
Champion

I think you should be able to use TIME_PREFIX to tell Splunk where the timestamp is and %s to tell Splunk the format. These setting need to go in props.conf. If the input is coming from a universal forwarder, then the props config should be on your indexer likely.

TIME_PREFIX = "timeStamp":
TIME_FORMAT = %s

There was a similar answer here: https://answers.splunk.com/answers/111161/how-do-i-get-splunk-to-recognise-epoch-time.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...