Getting Data In

How do I determine if hosts are in the wrong timezone?

Jason
Motivator

I have finished bringing in any archive data that I need to. Now all data flowing into Splunk is be up-to-the-minute, whether it be from a forwarder, a network port, or a script.

How do I tell if certain hosts sending data to Splunk have incorrect timezones?

Tags (3)
0 Karma
1 Solution

Jason
Motivator

This is a very important question, as normalization of time is essential so you can have an accurate picture of what actually happened at a certain point in time.

(If you have a datasource in Central time, but your indexer is in Eastern and wasn't told otherwise, it will index the data as Eastern and events will show in Splunk as an hour behind other events that happened that minute!)

One method is to try running this search. Run it over a period that would capture at least one event from each host, such as a day. It compares _time (the epoch time version of the time extracted from the event, adjusted with any timezone in the event or in props.conf) to _indextime (the epoch time version of when Splunk indexed the event). If your data is all current, these should be only seconds off if a datasource is properly configured.

* | dedup host | eval timediff=(_time-_indextime) | eval HoursOff=round(timediff/3600) | search NOT HoursOff=0 | rename _indextime as IndexTime | eval Time=_time | table Time, IndexTime, HoursOff, host

Remember this checks Index Time against Time extracted from the event. It assumes the most recent event from each host is coming in at current time, and isn't archive data.

I'm going to get this into an App, along with a script to add the timezone into the source field for troublesome linux servers. At some point.

View solution in original post

Jason
Motivator

This is a very important question, as normalization of time is essential so you can have an accurate picture of what actually happened at a certain point in time.

(If you have a datasource in Central time, but your indexer is in Eastern and wasn't told otherwise, it will index the data as Eastern and events will show in Splunk as an hour behind other events that happened that minute!)

One method is to try running this search. Run it over a period that would capture at least one event from each host, such as a day. It compares _time (the epoch time version of the time extracted from the event, adjusted with any timezone in the event or in props.conf) to _indextime (the epoch time version of when Splunk indexed the event). If your data is all current, these should be only seconds off if a datasource is properly configured.

* | dedup host | eval timediff=(_time-_indextime) | eval HoursOff=round(timediff/3600) | search NOT HoursOff=0 | rename _indextime as IndexTime | eval Time=_time | table Time, IndexTime, HoursOff, host

Remember this checks Index Time against Time extracted from the event. It assumes the most recent event from each host is coming in at current time, and isn't archive data.

I'm going to get this into an App, along with a script to add the timezone into the source field for troublesome linux servers. At some point.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...