Getting Data In

How to parse an unusual timestamp format?

jpolson
New Member

Hi all,

I have have some inconsistent timestamp parsing issues that I believe are due to an incorrect TIME_FORMAT value in my props.conf file and I am hoping that someone may be able to clarify what I've done wrong here. I get timestamps ingested into my Splunk instance with a format like this:

2017-01-31T19:35:43.379Z

This is the TIME_FORMAT value I have been using:

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N

It is mostly consistent but sometimes appears to not get parsed. Splunk Support has only been able to suggest at the Z at the end might be the issue, and on review of the documentation I don't see a specific way to note that in the TIME_FORMAT string. Does anyone know how to structure TIME_FORMAT to properly capture this?
I am not able to alter this data in any way so I must work with this format as-is.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

The Z is specific to Zulu time, i.e. UTC, i.e. GMT. In that same time zone specifier spot, you might see CDT for Central US Daylight Savings Time.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z

Please post the _raw timestamp from a couple of the events that did not parse, and we can analyze if there is another issue.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If the 'Z' is always present in the timestamp then just include it in the format string.

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3NZ
---
If this reply helps you, Karma would be appreciated.
0 Karma

jpolson
New Member

Thanks, Rich. I tried that and several variants of the %3N bit at the end but they don't appear to have any effect -- my Splunk instance is ignoring it and ingesting events at "now". Do you have any idea why that might be? I am using a very high value for MAX_TIMESTAMP_LOOKAHEAD so I believe that is not the issue.

Are there any other props.conf config settings that might be interfering with this?

0 Karma

MuS
SplunkTrust
SplunkTrust

Ensure that you configure it on the parsing level, either HWF if available or IDX http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings and also make sure to restart the Splunk instance you configured.

Check using btool on the parsing instance to see if your props is applied correctly or if some other props.conf is taking precedence over it.

cheers, MuS

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...