Thanks for help so far... I have been using dbinspect command to try to better understand the process and I wondered if what I am seeing is normal.
Whilst some buckets seem to follow the exact pattern described... i.e. they get created as hot, get filled with 1-2 hours of data, move to warm and make their way through to be deleted. I also have multiple buckets (old and new) that span 50+ days. Including ones that are hot. Is this normal as I think these are the problem ones...
bucketId startEpoch endEpoch timespanDays eventCount hostCount modTime sizeOnDiskMB state
sourceX~1353~X-X-X-X-X 20/10/2016 15:36 11/12/2016 16:33 52.081192 5686665 1 12/12/2016:09:08:28 894.238281 warm
sourceX~1354~X-X-X-X-X 09/12/2016 23:50 10/12/2016 08:48 0.373391 5966823 1 12/10/2016:08:48:38 870.082031 warm
sourceX~1355~X-X-X-X-X 10/12/2016 08:50 10/12/2016 16:26 0.316701 5878314 1 12/10/2016:16:27:20 853.476563 warm
sourceX~1356~X-X-X-X-X 10/12/2016 16:27 11/12/2016 00:14 0.323762 5955055 1 12/11/2016:00:14:26 876.789063 warm
sourceX~1357~X-X-X-X-X 10/12/2016 18:25 11/12/2016 09:41 0.636377 5900307 1 12/11/2016:09:42:00 858.257813 warm
… … … … … … … … …
sourceX~1444~X-X-X-X-X 20/12/2016 07:05 20/12/2016 10:53 0.158715 5408170 1 12/20/2016:10:54:13 885.84375 warm
sourceX~1445~X-X-X-X-X 20/09/2016 19:22 20/12/2016 12:24 90.751366 3594043 1 12/20/2016:14:38:59 500.539063 hot
sourceX~1447~X-X-X-X-X 20/12/2016 11:20 20/12/2016 12:21 0.04206 5511911 1 12/20/2016:12:21:21 897.332031 warm
sourceX~1448~X-X-X-X-X 20/12/2016 12:22 20/12/2016 13:11 0.034294 4385224 1 12/20/2016:13:12:04 593.207031 warm
sourceX~1449~X-X-X-X-X 20/12/2016 12:24 20/12/2016 13:41 0.053623 2996650 1 12/20/2016:14:38:59 418.238281 hot
sourceX~1450~X-X-X-X-X 20/12/2016 13:41 20/12/2016 14:38 0.039861 4860534 1 12/20/2016:14:39:08 675.105469 hot
So in this scenario
sourceX~1354~X-X-X-X-X will be deleted next as it has the oldest endEpoch - and it was
sourceX~1353~X-X-X-X-X and sourceX~1445~X-X-X-X-X span 50-90 days. Why is this? Shouldn't all buckets be similar.
Q: Is this normal? Should I just set a timelimit in indexes.conf? Something like maxHotSpanSecs=3600 so each bucket is only 1 hour?
... View more