Getting Data In

How to edit props.conf to ignore timezone information?

jwhughes58
Contributor

I've got data with a timestamp that looks like this

[2017-07-06T16:32:38.977-07:00]

In props.conf I have this

TIME_PREFIX = ^\[
MAX_TIMESTAMP_LOOKAHEAD = 24
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 999999
NO_BINARY_CHECK = true
TZ = UTC

What I'm finding is even though I ignore the TZ information in the TIME_FORMAT, the offset is getting added to _time. Unfortunately the data source uses the TZ field to signify where the device is physically located, but the time is actually UTC.

Time:7/7/17 6:34:06.657 PM  
Raw Event:[2017-07-07T11:34:06.657-07:00] [OUD] [TRACE] [OUD-24641552] [PROTOCOL] [host: myhostname.org] [nwaddr: aaa.bbb.ccc.ddd] [tid: 115] [userId: uid] [ecid: c6e497c51cca7a85:-7e0b16c4:15cf320db73:-8000-000000000330e42a,0:22] [result: 0] [category: RES] [conn: 305] [op: 127674] [etime: 2] [nentries: 1] [msgID: 127675] SEARCH

Any ideas on how I can get around this? All the other timestamps worked using Time_FORMAT, but they also have a space between the time and the TZ.

TIA,
Joe

0 Karma
1 Solution

paulstout
Path Finder

Joe,

Because we know this timezone offset is incorrect in the source data as it is annotating the time in which the device is deployed rather than the timezone in which the event occurred, I used SED to strip it out in props.conf and kept the rest of your configuration. It worked on my local Splunk with your line of test data.

I added this just before TIME_PREFIX:

SEDCMD-notz = s/\-[0-9][0-9]:[0-9][0-9]\]/]/

View solution in original post

0 Karma

paulstout
Path Finder

Joe,

Because we know this timezone offset is incorrect in the source data as it is annotating the time in which the device is deployed rather than the timezone in which the event occurred, I used SED to strip it out in props.conf and kept the rest of your configuration. It worked on my local Splunk with your line of test data.

I added this just before TIME_PREFIX:

SEDCMD-notz = s/\-[0-9][0-9]:[0-9][0-9]\]/]/
0 Karma

jwhughes58
Contributor

Paul,

Thanks this was almost the solution. I found a typo in my props.conf that was throwing things off. I also found this

SEDCMD-movetz = s/(^.{24})/\1 /

works better. Instead of removing the TZ information it adds a space between the time information and the TZ. Then my TIME_FORMAT worked correctly.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

What is the timezone of the user doing the search? I see the offset between event time (without offset in raw event) and _time value is 5 hrs and this is correct if user's timezone is in CST/CDT. The _time value adjusts based on who's viewing the data.

0 Karma

jwhughes58
Contributor

I always set my TZ to UTC. The offset between event time and _time value is 7 hours. 11:34:06 plus 7 hours is 18:34:06 or 6:34:06 PM. All my other data feeds work, showing event time and _time value as the same. Just this one with no space seems to grab the offset.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...