Getting Data In

How can I index events with the real date while using the timestamp from the log?

ZimmermanC1
Explorer

I have a Splunk instance in a Development & Test lab that uses what we call "repeatable time" to test software updates against a known good checkpoint of data. During a test all of the server times are rolled back to March 16 2011. I have been struggling to figure out how to get Splunk to ignore the Date in the logged events and only care about the "time".

The logs are being monitored in a directory structure as follows /var/adm/splunk/2017_"today's_julian_date"/"server_name"/*.log

 Example Log: header,123,321,2011-03-16 17:35:36.035 +00:00,subject,.............

Effectively I am trying to get these events indexed with today's date but the Time from the log. so it would have:

  _time=2017-12-11 17:35:36.035

I have tried playing with the TIME_PREFIX & MAX_TIMESTAMP_LOOKAHEAD settings in the sourcetype but have not been successful.

 TIME_PREFIX = ^[^\s]+\s
 MAX_TIMESTAMP_LOOKAHEAD = 25

This is a critical issue for me to sort out so any help would be greatly appreciated.

0 Karma

ZimmermanC1
Explorer

Following some more research I was able to determine that the date portion of the timestamp was being generated based on the file name which was called ASCII.20110316....... So i created a script to rename each file to ACII.2017-12-13....... and it appears to now almost be working. The problem seems to be that Splunk is now timeshifting the logs one day into the future. I assume it has something to do with the timezone but I am not 100% sure.

Any other ideas would be greatly appreciated.

0 Karma

DalJeanis
Legend

You might want to check out the Splunk Event Gen app, which IIRC can take input data and modify the timestamp in the _raw data for you.

https://splunkbase.splunk.com/app/1924/

0 Karma
Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...