Splunk should understand the Z=Zulu if you are using these settings in your props.conf
file:
TIME_PREFIX = ^\d+\w+
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z
MAX_TIMESTAMP_LOOKAHEAD = 28
If it doesn't, though, you can add this and it will work:
TZ_ALIAS = Z=UTC
You can tell these are working be checking the date_zone
field; it should NOT say local
.
If it STILL isn't correct, then the thing generating the event is misconfigured or clock-skewed and lying about the times (don't laugh; it happens) and you might be able to use the TZ_ALIAS
setting to compensate until you can fix it.
This is not a search
problem, it is a forwarding
problem. You have not told the Splunk timestamping entity (the Heavy Forwarder if you are using one, or the Indexers if you are not) what TZ to use for the timestamps in these logs. Figure that out and add a TZ=
configuration for your sourcetype
in props.conf
and then the event will be properly searchable.
1) What timezone is the event in? I notice timezone isn't included in the timestamp, so Splunk can't identify the timezone of the event.
2) What timezone is the server forwarding the logs set to? That is the fallback if the above fails.
3) What timezone is the indexer collecting the logs set to? This is the final fallback. (And highly unlikely, as it's very rare that 2 fails)
The best want to handle this is to modify your log formatting to include timezone offset in the timestamp. That way Splunk can parse it and will automatically handle it appropriately. If you can't do that, make sure the system doing the forwarding is configured appropriately, as that is the fallback.
If your syslog server is collecting data from multiple hosts in multiple timezones, things get more complicated. You're going to need to create props.conf stanzas which identify proper timezone (using the TZ=
setting) either by source, sourcetype, or hostname. (props.conf documentation)
Actually, the timestamp shown does have TZ information. The Z at the end of the timestamp means "Zulu time" or UTC. Splunk should understand this, but I suspect it's having problems with the subsecond resolution (do you really need 6 digits of subsecond precision?!?). Look in etc/datetime.xml and you'll see the automatic extractions.
Yes it was the subsecond percision. 3 digits works.