Using maxHotSpanSecs of 1 hour or 1 day generated too many buckets

[yannK]

Following this advice, I decided to rotate my hot buckets every hour. (each bucket should contains only 1 hour of data, without overlapping 2 hours)

But I discovered that at the hour many buckets are being created. I have 3 hot buckets and they close and new open all the time.
This seems to be because I have some servers with a time drift, so I was receiving events from different hours, over more than 3 hours (maxHotBuckets=3)

maxHotSpanSecs =
* Upper bound of timespan of hot/warm buckets in seconds.
* Defaults to 7776000 seconds (90 days).
* NOTE: If you set this too small, you can get an explosion of hot/warm
buckets in the filesystem.
* If you set this parameter to less than 3600, it will be automatically reset to
3600, which will then activate snapping behavior (see below).
* This is an advanced parameter that should be set
with care and understanding of the characteristics of your data.
* If set to 3600 (1 hour), or 86400 (1 day), becomes also the lower bound
of hot bucket timespans. Further, snapping behavior (i.e. ohSnap)
is activated, whereby hot bucket boundaries will be set at exactly the hour
or day mark, relative to local midnight.
* Highest legal value is 4294967295

[bbeard]

Your example for 24h (86399) looks correct. If you set to 3599 though (one hour less one second) you will invoke: If you set this parameter to less than 3600, it will be automatically reset to 3600, which will then activate snapping behavior (see below). You want to use at least 3601.

[yannK · Accepted solution]

My workaround was to use values for maxHotSpanSecs different from 3600 (1 hour), or 86400 (1 day).

Instead I used one hourless 1 second, or one day less 1 second.
Then my bucket span became a flexible rolling window, instead of a fixed window.

example for ~ 24h buckets
maxHotSpanSecs=86399