Why is there a one hour difference between event timestamps and index time of my data?

[omuelle1]

Hi Splunksters,

I am having an issue with the time the data is being indexed and the actual events being exactly one hour off.
I have tried to change timezones and so on, but so far I had to no luck.

I checked the time on the actual Linux box and it looks fine to me as well.

As you can see in the screenshot: the event is from 7:58 AM but data is being indexed at 8:58 AM. When I check the actual log file, this event is exactly one hour old as well, so I am always one hour behind in getting my data.

I also ran this search below and found that the index is exactly one hour behind:

Can anyone guide me in the right direction? Do I have to change the timezone on my forwarder in the props.conf?

Thank you so much,

Oliver

[woodcock · Accepted solution]

Splunk is interpreting the timestamp inside of your events wrong (1 hour in the future). You have 2 options to fix this:

1: If the agent generating events has a clock/timestamp/timezone problem, then fix it there.
2: If the timestamps are "correct", you need to tell Splunk what timezone to use to interpret the timestamps.

Assuming the latter, you have a few options:

1: You can update the agent generating the events and have it insert the TZ directly into the timestamp (this bloats the events and consumes license but it is the most foolproof).
2: You can configure a TZ value that allows splunk to adjust its interpretation of the time.

Assuming the latter, you just put something like this in props.conf and restart all splunk instances where you deploy it:

[host::172\.0\.0\.1]
TZ = US/Central

[maciep]

I believe those props settings are processed during the parsing phase. So if your forwarder is actually a heavy forwarder, then that's where you'd make the changes. If it's a universal forwarder, then you'll want to move those changes to your indexer.

Where do I configure my Splunk Settings?

[woodcock]

The props.conf should be in the same directory where the associated inputs.conf exists and you should do splunk restart. This will fix all events that are processed from that point on but do not reassess until 1 hour after you restart the indexers because you will have to wait for all of the events that you "sent to the future" to trickle back through to "now".

[omuelle1]

I changed it to Eastern timzone, since that is the TZ our servers are in.

We are using a Universal Forwarder on this app.

My props.conf on the forwarder looks like this:

[mstris]
# A performance tweak is to disable SHOULD_LINEMERGE and then set the
# LINE_BREAKER to "line ending characters coming before a new time stamp"
# (note the direct link of the TIME_FORMAT to the regex of LINE_BREAKER).
LINE_BREAKER = ([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}-\d{2}:\d{2}
SHOULD_LINEMERGE = False
# 10000 is default, should be set on a case by case basis
TRUNCATE = 10000
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 29
TZ = US/Eastern

Leaving PUNCT enabled can impact indexing performance. Customers can
Comment this line if they need to use PUNCT (e.g. security use cases)
ANNOTATE_PUNCT = false
Inline field extraction
EXTRACT-connection_id = (?i)(connectionid|conn_id|callid)\=(?\d{6})
Field extraction using a transforms.conf
 REPORT-auto_kv_for_all_fields = auto_kv_for_all_fields

If the data does not have nice key=value pairs, (or some other readily
machine parseable format, like JSON or XML), set KV_MODE = none so that
Splunk doesn't spin its wheels on attempting to look for key = value
pairs which don't exist.
KV_MODE = auto

SEDCMD-RemoveComments = s/^#.*$//
SEDCMD-RemoveInfoLines = s/.*\[Info\].*//
SEDCMD-RemoveBadObject = s/.*The object server is unable to find an object with ID '00000000000000000000000000000000'.*//
EXTRACT-pid = (?i)\[PID:(?P[^\]]+)
EXTRACT-thread_id = (?i)\[THR:(?P[^\]]+)
EXTRACT-service = (?i)^(?:[^\[]*\[){4}(?P[^\]]+)
EXTRACT-log_level = (?i)^(?:[^\[]*\[){5}(?P[^\]]+)
EXTRACT-message = (?i)(?:Error|Debug|Warning|Info)\]\s*(?P.*)

I have US/Eastern now in my forwarder and also see for props.conf on the indexers for this app, however, I am still seeing the same issue with the time stamp in SPLUNK being one our in the future:

10/1/15 
9:17:08.899 AM
2015-10-01 08:17:08.899-05:00 [HOST:.corp.rndc-usa.com][PID:61838][THR:1504118528][Kernel][Info] Contract Request Succeeded: Source=MsiWorkingSet::MCMPackBSTR, Handle=42331, Size=85450752, ContractedTotal=1735479296, AvailableTotal=135937250158

Where else could I change props.conf that is related to this index? Sorry but I am very new to Splunk..

[omuelle1]

Thank you for answer.

So I would need to change my props.conf on the forwarder where I am getting the data from, correct? Because I did this and put TZ=US/Central in there and I still keep getting the same time.

this is from my props.conf on the forwarder where I am getting the data from:

TRUNCATE = 10000
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%3NZ
MAX_TIMESTAMP_LOOKAHEAD = 29
TZ = US/Central

Oliver

[woodcock]

First of all, I do not know if US/Central is the correct TZ; I had to use something as an example in my answer and I used my TZ. You need to figure out what TZ value is correct for your forwarder.

Also, you did not show your stanza header in your props.conf configuration so I will delay commenting on it.

Did you restart splunk on the forwarder after you made this change?

[omuelle1]

This is the result to that search:

Thank you so much for your help!

[woodcock]

There is no doubt about it: your events are being timestamped 1 hour "into the future". I will post a followup answer.

[omuelle1]

Sorry the first picture should be this: