Duplicate of : How to make Splunk guess the correct date if only time is given?
Splunk indexes times in our source and gets the date wrong with no apparent way to override the default.
In our sourcetype;
- The event contains a timestamp but not a datestamp - fails rule 1 & 2 & 3 below
example: 23:59:59.441 Trc interesting event data 00:00:00.742 Trc additional interesting event data
example: -rw-r--r-- 1 user user 104857706 Jul 20 07:03 trc_urs_mm_p.20180719_203048_911.log
For our needs, Rule 5 or Rule 6 would be preferable. This question is: does anyone have a workaround or override that can be implemented at indextime?
I found references here:
http://docs.splunk.com/Documentation/Splunk/7.1.2/Data/Configuretimestamprecognition
refers to $SPLUNK_HOME/etc/datetime.xml
Reading this file I can't see how it would help unless there's a special syntax that can be used to manipulate the guessed date in some way.
Note:
- We cannot change the log format as it's proprietary software
- We could prepend every line of the log with the date by streaming the log in real time through a script but this is ugly, will temporarily double the space required for the logs (minor issue) will increase i/o wait and cpu time (moderate issue) and is another process that could go wrong.
- We can try to force the application to start a new log each day at midnight (it normally creates a new log once current log reaches a specified size).
If there's no clever Splunk solution available I could ask our team to request a feature but I won't hold my breath to have that happen. Ideally Splunk would let us decide the precedence of the rules per sourcetype, or allow an override along the lines of "If the date is guessed - then if the current event date/time is more than 23.5 hours (configurable) less than the previous date/time, add 1 day the guessed date for this event.
In other words, 0:00:00 is the next day when the previous event had timestamp like 23:59:59. Currently in our situation Splunk gets it wrong.
http://docs.splunk.com/Documentation/Splunk/6.5.2/Data/HowSplunkextractstimestamps
How Splunk software assigns timestamps
Splunk software uses the following precedence rules to assign timestamps to events:
TIME_FORMAT
, if provided. You configure the TIME_FORMAT
attribute in props.conf.TIME_FORMAT
was configured for the data, Splunk software attempts to automatically identify a time or date in the event itself. It uses the source type of the event (which includes TIME_FORMAT
information) to try to find the timestamp.Splunk software can extract only dates from a source, not times. If you need to extract a time from a source, use a transform. See Create custom fields at index time.
Have you tried just TIME_FORMAT = %H:%M:%S.%3N
in your props.conf? Splunk extract the correct time and will apply the date from the current system time which would be fine if you're indexing live files.