How does Splunk handle timestamps from different timezones when it doesn't know the offset?

[hlarimer]

How does Splunk handle timestamps from different timezones when it doesn't know offset? I'm seeing different behaviors on logs coming in from firewalls (all Palo Alto's) from different timezones.....

For example, I have a FW that is a timezone away from the Splunk Forwarder where it sends its logs. When I look at the logs I see that Splunk is changing the logs to the timezone local to the Splunk Forwarder. But I have other FWs that are a few timezones away and Splunk is not changing their timestamps.

So when doing a search across all FW's for a something that happened an hour ago, I get results from some FW's for things that didn't necessarily happen an hour ago.

Is there any reasoning behind this?

[musskopf]

Could pls paste here raw events from both firewall that happen at similar time?

Normally Splunk will convert to local time zone if no time zone has been provided. The only exception if I'm not wrong is if the timestamp is presented as epoch seconds Splunk will interpret as being in UTC (as far I remember).

[hlarimer]

The logs I am seeing are from Palo Alto's and the documentation is asking to use "no_appending_timestamp = true" for inputs.conf. I'm wondering how this is affecting the logs?

We are finding a couple inconsistencies here but I think the next step is to figure out how to handle firewalls that are located in different geographical locations. If they are all reporting their logs in their local time and I do a search to try to correlate something that could be happening across firewalls (like a virus outbreak trying to communicate out), then I'm not going to see events from some firewalls due to the timestamps.

But if they are all timestamped by the indexer then old logs that are coming in (like after a network outage) will be timestamped incorrectly.

Am I over thinking this?

[kml_uvce · Accepted solution]

If i understood your problem...

read this..
http://docs.splunk.com/Documentation/Splunk/4.1.8/admin/ApplyTimezoneOffsetstotimestamps#zoneinfo_.2...
you need to configure in indexer
you can find entries of TZ at: http://en.wikipedia.org/wiki/List_of_zoneinfo_timezones

[hlarimer]

What we have decided to do is very close to this solution. We have decided to set all Firewalls to UTC and then set the props.conf on the indexers for the source that corresponds to those firewalls to TZ = UTC. This way we don't have to worry about setting the TZ offset for each FW, but instead can have it work for all FW's globally as long as they are set to UTC.

Thanks for your help.