I have a Splunk instance in a Development & Test lab that uses what we call "repeatable time" to test software updates against a known good checkpoint of data. During a test all of the server times are rolled back to March 16 2011. I have been struggling to figure out how to get Splunk to ignore the Date in the logged events and only care about the "time".
The logs are being monitored in a directory structure as follows /var/adm/splunk/2017_"today's_julian_date"/"server_name"/*.log
Example Log: header,123,321,2011-03-16 17:35:36.035 +00:00,subject,.............
Effectively I am trying to get these events indexed with today's date but the Time from the log. so it would have:
_time=2017-12-11 17:35:36.035
I have tried playing with the TIME_PREFIX & MAX_TIMESTAMP_LOOKAHEAD settings in the sourcetype but have not been successful.
TIME_PREFIX = ^[^\s]+\s
MAX_TIMESTAMP_LOOKAHEAD = 25
This is a critical issue for me to sort out so any help would be greatly appreciated.
Following some more research I was able to determine that the date portion of the timestamp was being generated based on the file name which was called ASCII.20110316....... So i created a script to rename each file to ACII.2017-12-13....... and it appears to now almost be working. The problem seems to be that Splunk is now timeshifting the logs one day into the future. I assume it has something to do with the timezone but I am not 100% sure.
Any other ideas would be greatly appreciated.
You might want to check out the Splunk Event Gen app, which IIRC can take input data and modify the timestamp in the _raw data for you.