Configuring TimeStamp Format

[thomastaylor]

Hello everyone!

Currently, we're running into a problem configuring our timestamp for a source type. After browsing a few questions on Splunk Answers, we are still not able to correctly implement the timestamp.

When we leave the extraction on "Auto", it does not extract the timezone from our logs:

2018-05-23 08:49:59.266814 - Toaster 375790 - INFO - Boot up sequence begin
2018-05-23 08:46:23.836273 - Toaster 450623 - INFO - Loading: Network Lib

TIME_FORMAT =
%Y-%m-%d %H:%M:%S.%6N

We left the TIME_PREFIX empty because the timestamp is at the beginning of our log.

We have also attempted to use TIME_PREFIX = ^

Remaining Configuration:
MAX_TIMESTAMP_LOOKAHEAD = 50
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TZ = America/New_York
category = Application
disabled = false
pulldown_type = true

[thomastaylor · Accepted solution]

Thank you to everyone who answered this question! I should have included more details, but I now know what the solution was. This event was received by the HTTP Event Collector. I was trying to parse the raw event that was sent.

By referencing this question, https://answers.splunk.com/answers/312321/can-we-process-the-timestamp-in-an-event-sent-to-t.html, you can see that, for time to be extracted properly, the "time" keyword must be placed in the post call.

[skoelpin]

You should include TIME_PREFIX=^ and you should change your MAX_TIMESTAMP_LOOKAHEAD to something more reasonable like 28 rather than 1000

[thomastaylor]

We have tried this as well. We just tried it again to make sure and it still did not yield the correct result.

[skoelpin]

The info I provided wont fix the TZ issue, but rather take some of the load off your indexers.. As for TZ, are the servers with the UF's installed in the same TZ as your indexers? What's your end goal? Have you tried change the TZ in the Splunk UI?

[thomastaylor]

Thanks for the response!

Right now, we are using the Splunk Cloud client as a proof of concept. We have not figured out a way to configure Splunk to pull the timestamp correctly from our log. We have changed the timezone in the UI to match the timezone from our log source.

The timestamp that shows on the Splunk search is the time that the log was received from the server, but not the time when the log was generated.

[adonio]

where is the timezone in the data?
your format looks ok,
can you share the full relevant props.conf?
will recommend to create a sourcetype regardless and not leave on auto

[thomastaylor]

We have actually already created the source type named "Kitchen Appliances". We have updated the post with the full props.conf that we can view from the cloud client.

[FrankVl]

Can you provide some details on how you have deployed that configuration? What does the rest of your props.conf look like? On which instance is it deployed (and what is your architecture)?

Is that sourcetype the original sourecetype from inputs.conf, or are you overriding the sourcetype using transforms,conf?

[thomastaylor]

We are currently using the cloud and cannot access the props.conf. We have created our own source type named Kitchen Appliances.

Remaining Configuration:

MAX_TIMESTAMP_LOOKAHEAD = 1000
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = true
TZ = America/New_York
category = Application
disabled = false
pulldown_type = true