Getting Data In

How to show search times in local time vs. future when using syslog message with UTC time stamp

simpkins1958
Contributor

Have syslog message with time stamp:

<134>1 2014-11-25T18:22:48.720252Z EMM-JimS-01

Splunk search is not showing those times in local time. It is showing times in the future.

Can search be adjusted to show local times from syslog UTC times?

Tags (3)
0 Karma

woodcock
Esteemed Legend

Splunk should understand the Z=Zulu if you are using these settings in your props.conf file:

TIME_PREFIX = ^\d+\w+
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z
MAX_TIMESTAMP_LOOKAHEAD = 28

If it doesn't, though, you can add this and it will work:

TZ_ALIAS = Z=UTC

You can tell these are working be checking the date_zone field; it should NOT say local.

If it STILL isn't correct, then the thing generating the event is misconfigured or clock-skewed and lying about the times (don't laugh; it happens) and you might be able to use the TZ_ALIAS setting to compensate until you can fix it.

0 Karma

woodcock
Esteemed Legend

This is not a search problem, it is a forwarding problem. You have not told the Splunk timestamping entity (the Heavy Forwarder if you are using one, or the Indexers if you are not) what TZ to use for the timestamps in these logs. Figure that out and add a TZ= configuration for your sourcetype in props.conf and then the event will be properly searchable.

0 Karma

emiller42
Motivator

1) What timezone is the event in? I notice timezone isn't included in the timestamp, so Splunk can't identify the timezone of the event.
2) What timezone is the server forwarding the logs set to? That is the fallback if the above fails.
3) What timezone is the indexer collecting the logs set to? This is the final fallback. (And highly unlikely, as it's very rare that 2 fails)

The best want to handle this is to modify your log formatting to include timezone offset in the timestamp. That way Splunk can parse it and will automatically handle it appropriately. If you can't do that, make sure the system doing the forwarding is configured appropriately, as that is the fallback.

If your syslog server is collecting data from multiple hosts in multiple timezones, things get more complicated. You're going to need to create props.conf stanzas which identify proper timezone (using the TZ= setting) either by source, sourcetype, or hostname. (props.conf documentation)

0 Karma

tetlowgm
Engager

Actually, the timestamp shown does have TZ information. The Z at the end of the timestamp means "Zulu time" or UTC. Splunk should understand this, but I suspect it's having problems with the subsecond resolution (do you really need 6 digits of subsecond precision?!?). Look in etc/datetime.xml and you'll see the automatic extractions.

0 Karma

simpkins1958
Contributor

Yes it was the subsecond percision. 3 digits works.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...