Please take a look at this post https://answers.splunk.com/answers/235558/wrong-datetime-conversion-from-epoch.html I believe it is probably the same issue.... (I cannot validate, as I cannot view the posted screenshots from here). I believe there is a bug in the splunk db connect app when converting epoch timestamps to human readable form. If left in epoch format, splunk indexes these without issue. ... View more

I have changed my Splunk DB Connect - DB Input to use 'epochtime' format as its conversion (rather than the default human readable format.) This problem then does not occur. This seems to be a reasonable workaround (for me at least). ... View more

I've done more investigation, and actually found that this happens on the last few days on each year within the data. My assumption therefore is that the splunk DB connect app is calculating the year incorrectly from the epoch format timestamp before indexing. (As the event string itself has the converted date, which is incorrect - and the splunk event time always matches this converted date). e.g. Event Time 31/12/2014 15:18:27.000 Event Data "2014-12-31 15:18:27" CHANGEID=1770752, CHANGETS=1388503107000 Where 1388503107000 should be GMT: Tue, 31 Dec 2013 15:18:27 GMT ... View more

UPDATE (workaround) I have changed my Splunk DB Connect - DB Input to use 'epochtime' format as its conversion (rather than the default human readable format.) This problem then does not occur. This seems to be a reasonable workaround (for me at least). Original Response I've just had exactly the same problem... reading in data from 2011-2015, the ONLY date effected was 29th December 2014, where ALL epoch times for this date have been converted to have the year 2015.... I expect this is a problem with the "splunk db connect" app rather than splunk itself. ... View more

I have exactly the same issue (using log4j with buffered io and immediate flush of false) Splunk (universal forwarder) is picking up some of the events before they have been fully logged (they can be truncated mid line before enough data accumulates and flushes the remainder of the event from the buffer to the file system). When the remaining log data is written and sent to the indexer, it is indexed as a separate event. (regardless of any SHOULD_LINEMERGE settings etc.) My guess is that there is some sort of timeout for multi line events to complete? ... View more