Can we override the guessed date for events that have no datestamp ?

[pclewis]

Duplicate of : How to make Splunk guess the correct date if only time is given?
Splunk indexes times in our source and gets the date wrong with no apparent way to override the default.

In our sourcetype;
- The event contains a timestamp but not a datestamp - fails rule 1 & 2 & 3 below

    example: 
    23:59:59.441 Trc interesting event data
    00:00:00.742 Trc additional interesting event data

• The filename of the source contains the date and time that the new log (source) was created - success rule 4?
• The last modified time is updated correctly

    example:
    -rw-r--r-- 1 user user 104857706 Jul 20 07:03 trc_urs_mm_p.20180719_203048_911.log

For our needs, Rule 5 or Rule 6 would be preferable. This question is: does anyone have a workaround or override that can be implemented at indextime?

I found references here:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Configuretimestamprecognition
refers to $SPLUNK_HOME/etc/datetime.xml
Reading this file I can't see how it would help unless there's a special syntax that can be used to manipulate the guessed date in some way.

Note:
- We cannot change the log format as it's proprietary software
- We could prepend every line of the log with the date by streaming the log in real time through a script but this is ugly, will temporarily double the space required for the logs (minor issue) will increase i/o wait and cpu time (moderate issue) and is another process that could go wrong.
- We can try to force the application to start a new log each day at midnight (it normally creates a new log once current log reaches a specified size).

If there's no clever Splunk solution available I could ask our team to request a feature but I won't hold my breath to have that happen. Ideally Splunk would let us decide the precedence of the rules per sourcetype, or allow an override along the lines of "If the date is guessed - then if the current event date/time is more than 23.5 hours (configurable) less than the previous date/time, add 1 day the guessed date for this event.
In other words, 0:00:00 is the next day when the previous event had timestamp like 23:59:59. Currently in our situation Splunk gets it wrong.

http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/HowSplunkextractstimestamps
How Splunk software assigns timestamps
Splunk software uses the following precedence rules to assign timestamps to events:

1. It looks for a time or date in the event itself using an explicit TIME_FORMAT, if provided. You configure the TIME_FORMAT attribute in props.conf.
2. If no TIME_FORMAT was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT information) to try to find the timestamp.
3. If an event has a time and date, but not a year, Splunk software determines the year, as described in How Splunk software determines timestamps with no year, and builds the timestamp from that.
4. If no events in a source have a date, Splunk software tries to find a date in the source name or file name. Time of day is not identified in filenames. (This requires that the events have a time, even though they don't have a date.)
5. For file sources, if no date can be identified in the file name, Splunk software uses the file modification time.
6. As a last resort, Splunk software sets the timestamp to the current system time when indexing each event.

Splunk software can extract only dates from a source, not times. If you need to extract a time from a source, use a transform. See Create custom fields at index time.

[jplumsdaine22]

Have you tried just TIME_FORMAT = %H:%M:%S.%3N in your props.conf? Splunk extract the correct time and will apply the date from the current system time which would be fine if you're indexing live files.