I am trying to index data for future events (reservation searches), to (line)chart when people are looking to travel. These dates range from a year ago to a few years in the future. It seems to be giving future events a timestamp of 3 days from the time the data was loaded.
Is there any trick to this?
So it looks like there is a timestamp confusion here. The timestamp that Splunk creates when it indexes data is ideally drawn from the time the event was written to the log file/database/etc. In your use case here it seems that date fields you are working with are separate from the timestamp.
So to make up an example:
June 7, 2012 15:37 EDT Requested reservation SFO - LAX out: Sept 1 2012 return: Sept 10 2012
In my made up example here "June 7 2012 15:37 EDT" would be the timestamp. It's the time and date i attempted to make the reservation. The subsequent dates Sept 1 and Sept 10 are not the timestamp, they are just date based fields contained in the data. You want to run reports on the date fields not the timestamp.
Not knowing exactly what your data looks like I can't give you specifics on how to let Splunk know the difference between a date field and the actual timestamp of the event. BUT I can point you to this doc page:
http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/HowSplunkextractstimestamps
For your particular use case it may make sense to disable timestamping all together:
Then you can extract out the reservation dates as separate fields and run your reports against those fields.
http://docs.splunk.com/Documentation/Splunk/4.3.2/Knowledge/Addfieldsatsearchtime
Also the reason no dates beyond 3 days in the future would index is because Splunk is set to automatically ignore future timestamps beyond 3 days.
See the Max_Days_Hence information here: http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf
If you are using Splunk 4.3 the Preview feature in Manager can be VERY helpful when you need to teach Splunk where to look and where NOT to look for a timestamp.
Thanks. Disabling timestamping did the trick and the other references helped with understanding why. Below is my working search after making the change.
index=CALENDAR_SEARCHES | convert timeformat="%Y-%m-%d %T" mktime(CALENDAR_DATE) | fieldformat CALENDAR_DATE=strftime(CALENDAR_DATE,"%Y-%m-%d %T") | makecontinuous CALENDAR_DATE span=1m | chart avg(COUNT) by CALENDAR_DATE
It's the default setting of a key called MAX_DAYS_HENCE. Splunk is hitting up against a failsafe here. You should weaken it just for that particular sourcetype.
here are the docs:
http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition
I guess I was assuming that these timestamps were the "correct" times that you wanted the events indexed at. 😃 Sounds like that was not the case.