Dashboards & Visualizations

future events on timechart

s8ea8n
Engager

I am trying to index data for future events (reservation searches), to (line)chart when people are looking to travel. These dates range from a year ago to a few years in the future. It seems to be giving future events a timestamp of 3 days from the time the data was loaded.

Is there any trick to this?

Tags (1)

DrewO
Splunk Employee
Splunk Employee

So it looks like there is a timestamp confusion here. The timestamp that Splunk creates when it indexes data is ideally drawn from the time the event was written to the log file/database/etc. In your use case here it seems that date fields you are working with are separate from the timestamp.

So to make up an example:

June 7, 2012 15:37 EDT Requested reservation SFO - LAX out: Sept 1 2012 return: Sept 10 2012

In my made up example here "June 7 2012 15:37 EDT" would be the timestamp. It's the time and date i attempted to make the reservation. The subsequent dates Sept 1 and Sept 10 are not the timestamp, they are just date based fields contained in the data. You want to run reports on the date fields not the timestamp.

Not knowing exactly what your data looks like I can't give you specifics on how to let Splunk know the difference between a date field and the actual timestamp of the event. BUT I can point you to this doc page:

http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/HowSplunkextractstimestamps

For your particular use case it may make sense to disable timestamping all together:

http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/TuneTimestampExtractionForBetterIndexingPerfo...

Then you can extract out the reservation dates as separate fields and run your reports against those fields.

http://docs.splunk.com/Documentation/Splunk/4.3.2/Knowledge/Addfieldsatsearchtime

Also the reason no dates beyond 3 days in the future would index is because Splunk is set to automatically ignore future timestamps beyond 3 days.

See the Max_Days_Hence information here: http://docs.splunk.com/Documentation/Splunk/latest/admin/propsconf

If you are using Splunk 4.3 the Preview feature in Manager can be VERY helpful when you need to teach Splunk where to look and where NOT to look for a timestamp.

s8ea8n
Engager

Thanks. Disabling timestamping did the trick and the other references helped with understanding why. Below is my working search after making the change.

index=CALENDAR_SEARCHES | convert timeformat="%Y-%m-%d %T" mktime(CALENDAR_DATE) | fieldformat CALENDAR_DATE=strftime(CALENDAR_DATE,"%Y-%m-%d %T") | makecontinuous CALENDAR_DATE span=1m | chart avg(COUNT) by CALENDAR_DATE

0 Karma

sideview
SplunkTrust
SplunkTrust

It's the default setting of a key called MAX_DAYS_HENCE. Splunk is hitting up against a failsafe here. You should weaken it just for that particular sourcetype.

here are the docs:

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition

0 Karma

sideview
SplunkTrust
SplunkTrust

I guess I was assuming that these timestamps were the "correct" times that you wanted the events indexed at. 😃 Sounds like that was not the case.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...