Linux Syslog and Timezone Offset

[BP9906]

Hello,
We have varying application servers that have different time zones set based on the customer's timezone location. This helps with a variety of things like maintenance hours and setting things based on the customer's local time.

The short story is that our servers are named according to where they are physically located, not customer location, thus we have no easy way to maps machines to timezones. Searching around, it seems like the Universal Forwarder on all the servers we have cannot be told what timezone they are in, but that I must inform the indexer(s) receiving data from the forwarder via props.conf entry:

Example:

[host::nyapplicationcustomer1*]
TZ = US/Central

Is there no easier way to tell the forwarder that its default timezone and UTC offset should be based on its local machine timezone setting?

For logs that by default have no timezone offset, this raises a larger issue and I'm surprised the forwarder isnt smart enough to know about the timezone setting for the server it is looking at the logs for.

Thank you,
Brian

[BP9906 · Accepted solution]

Thought I'd add that I have a scripted input to collect Linux timezone:
TimeZone=$(grep -i "zone=" /etc/sysconfig/clock | awk -F= '{print $2}' | sed 's/"//g');
Then I have a splunk search to let me dump the info and format it like props.conf

[host::]
TZ =

......

Its a pain, but I can easily update the list and recycle the indexers regularly.

[the_wolverine]

You can set offset at the forwarder level if you can convert your your forwarders to Heavyweight forwarders. This might drastically change performance so you'd want to test this out.

Or, if you can get the remote hosts to send a TZ in the timestamp, Splunk will automatically recognized the TZ and will offset it correctly at the server.

[BP9906]

Yep, makes sense. Thanks. I've had to add most of the TZ offsets to props.conf

[the_wolverine]

This is at the application/system level for the remote host. Unfortunately we have encountered legacy systems which don't support this. In that case you would have to offset if their TZ does not match your Splunk server TZ.

[BP9906]

How would you get the remote hosts to send a TZ in the timestamp?