How to make the time stamp of logs default to _indextime?

[awedmondson]

Hi,
Is there a way to have the time stamp of logs to default to the _indextime? I have noticed that a few events from a particular index are incorrect, and would like the time to automatically default to _indextime rather than _time.
When I do a search with eval indextime=strftime(_indextime,"%y/%m/%d %H:%M:%S")

It is correct, but how to do this automatically, so that _time reported from this index is really _indextime?

thanks
Allen

[marycordova]

see fully analyzed problem and solution here: https://answers.splunk.com/answers/678655/how-to-trigger-alerts-when-indextime-time.html

[marycordova]

actually, using this config i am still showing a 7-8 second difference between _time and _indextime

the problem is that if i have an event that is timestamped at the :57 second but is not indexed until the :03 second and my search runs at the :00 second then the event is not actually present on disk to be searched until after the search has run and will not be found during the next search which starts at the :00 second 5 minutes later

does anyone know how to truly set _time to _indextime using either props or transforms? it seems like in the order of field operations the only thing that would work is to use and EXTRACT in props then a REPORT and transform on the extracted field however it does not seem that an EXTRACT works on _indextime

[sudosplunk]

This is because of latency in indexing. Alternatively, you can use DATETIME_CONFIG=NONE to leave the event time set to whatever time was selected by the input layer (for ex. Universal Forwarder) or if you have timestamps in your events, you can use TIME_PREFIX, MAX_TIMESTAMP_LOOKAHEAD and TIME_FORMAT settings to extract timestamp from events as described here.

[marycordova]

this does not address the problem at all and actually makes it worse

what if the extracted time stamp is 15 minutes earlier than the indexing time or the system time?

i just got off the phone with support, there is a solution, i will post the answer shortly

[sudosplunk]

I misunderstood the question and am glad you were able to find solution!

[marycordova]

Problem definition: there are 3 possible "times" associated with an event and this can cause events to be missed in scheduled searches

Example: _time may have value 1 OR 2 but not 3 (_indextime)

the timestamp listed in the _raw event data (TIME_PREFIX or other config) = 0:4:58

the time the event is seen up by the forwarder (CURRENT) = 0:5:58

the time the event is written to disk (_indextime) = 0:6:03

if either time 1 or time 2 is not equal to _indextime your event can be missed by a search

for example if you have an event where CURRENT time is 0:5:58 seconds, timestamp from _raw is 0:4:58, and _indextime is 0:6:03 then a search running every minute -1m@m to @m will never see this particular event

for sourcetypes with various props configurations such as EXTRACT, EVAL, FIELDALIAS, LOOKUP, etc there can easily be differences of several seconds between _time and _index time meaning an event is not available on disk until after the time the scheduled search has run

Solution:

if the data set permits set _time to the following in props.conf: DATETIME_CONFIG = CURRENT

re-write -1m@m to @m searches to accommodate potential delays: -2m@m to -1m@m

you can be creative with searches that hourly or daily for example -61m@m to -1m@m

don't forget to adjust your cron schedule accordingly if necessary

[awedmondson]

Does DATETIME_CONFIG = CURRENT work in Splunk version 5.04?

[gfuente]

Yes it does

[cpetterborg · Accepted solution]

In the props.conf file for the source, use:

DATETIME_CONFIG = CURRENT

This will set the _time to the current time (index time). More information can be found here:

http://docs.splunk.com/Documentation/Splunk/6.2.0/Data/Configuretimestamprecognition

[marycordova]

after much debate...this is not an advisable solution, see the thread here: https://answers.splunk.com/answers/678655/how-to-trigger-alerts-when-indextime-time-1.html