Hi,
I have a log file with timestamp = time of saving the file thus the timestamp from splunk are all the same. I not want to extract the epoch time example "1495447178314" and make it as my timestamp. Is that possible?
I also tried to extract new field using regular expression \d{13}[^\}\}]*
but could not get anything when I applied it. Using https://regex101.com/ to test it out, somehow it looks like there were 2 matches. Can someone help? Maybe is it the way I do the \}\}
?
15199256 [EPS-log-dispatcher-9] INFO
1.24978294676695149906 - {"Log Header": "{"endpointKeyHash":null,"applicationToken":null,"headerVersion":null,"timestamp":null,"logSchemaVersion":null}", "Event": {"temperature":35,"timeStamp":1495447178314}}
I think you should be able to use TIME_PREFIX to tell Splunk where the timestamp is and %s to tell Splunk the format. These setting need to go in props.conf. If the input is coming from a universal forwarder, then the props config should be on your indexer likely.
TIME_PREFIX = "timeStamp":
TIME_FORMAT = %s
There was a similar answer here: https://answers.splunk.com/answers/111161/how-do-i-get-splunk-to-recognise-epoch-time.html
Hi,
I managed to make the time format from Epoch to human readable but I can't really get the millisecond out.
Example timeStamp":1495447178314
From Splunk it converted to "5/22/17 5:59:38.000 PM" but from https://www.epochconverter.com/, it is showing
May 22, 2017 5:59:38.314 PM
Reference document: http://docs.splunk.com/Documentation/Splunk/6.0/Data/Configuretimestamprecognition, .%3N should show the milliseconds.
Another weird thing is, I had to use %s%s.%3N to show the time in my time zone. If I use %s.%3N, it will show all my time to be "12/31/99 11:59:59.999 PM". Am I supposed to use just %s.%3N?
i'm having a little trouble reading your screenshots. But those settings should be in props.conf for the sourcetype of the log you're ingesting. I see you referenced the stanza for custom_log, but what about for the IoT Temperature sourcetype? The timestamp settings need to be applied there.
Actually, I didn't know how and where to edit the props.conf ...
I think I used the UI to change the timestamp format and time prefix and saved a new sourcetype? Is this ok?
So are you just uploading the file in the GUI over an over? Not sure what your env looks like, how you plan to ingest this data, etc. It doesn't matter necessarily what you call the sourcetype, but that you are using the same sourcetype when you search for the data as when you ingest it.
So is the timestamp correct in Splunk yet? Maybe not to the ms, but is it no longer 6/10? I'm having trouble following along with all of these posts.
The latest I got the the post below with EpochCOnverter screenshot. Actually, the above screenshots are all different. I think we can skip the above screenshots now since I managed to get the timestamp now but with the milliseconds still having some issues.
So the timestamp of the splunk event is correct (minus milliseconds)? You're not just extracting the timestamp to a new field, right? There is a big difference, so want to be sure splunk timestamping is working as expected.
yup, timestamp working minus milliseconds and I'm no longer extracting it as a new field.
Ok. I'm not sure about the millseconds part, maybe %s%3N?
oh....you were right...example has a dit and I just follow but my data did not have. Now it shows the milliseconds...thanks.
Do I just add to 1st 2 rows of props.conf file below [custom_log]?
Currently my props.conf file is as follows:
[custom_log]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Application
description = For IoT Demo
disabled = false
maxDist = 75
pulldown_type = true
I tried to upload the data again but it still shows the file saved time instead of the timestamp. Is there other ways such as extracting it as another field?
I think you should be able to use TIME_PREFIX to tell Splunk where the timestamp is and %s to tell Splunk the format. These setting need to go in props.conf. If the input is coming from a universal forwarder, then the props config should be on your indexer likely.
TIME_PREFIX = "timeStamp":
TIME_FORMAT = %s
There was a similar answer here: https://answers.splunk.com/answers/111161/how-do-i-get-splunk-to-recognise-epoch-time.html