Order in which buckets roll from hot/warm -> cold

[forrest_whitche]

From: http://docs.splunk.com/Documentation/Splunk/4.3/admin/HowSplunkstoresindexes

"Once Splunk has created some maximum number of warm buckets, it begins to roll the warm buckets to cold based on their age. Always, the oldest warm bucket rolls to cold. Buckets continue to roll to cold as they age in this manner. After a set period of time, cold buckets roll to frozen, at which point they are either archived or deleted. By editing attributes in indexes.conf http://docs.splunk.com/Documentation/Splunk/4.3/admin/Indexesconf, you can specify the bucket aging policy http://docs.splunk.com/Documentation/Splunk/4.3/admin/Setaretirementandarchivingpolicy, which determines when a bucket moves from one stage to the next."

This is not what I'm seeing in the wild, e.g. Here are my hot+warm (local disk) and cold (NFS mount) directories (see below)

Observe that hot/warm includes buckets as old as Dec 20, while recent buckets in the colddb directory have been created at various times over the last month and then include the files that were moved when I first needed to move cold into NFS to avoid overflowing the local hot/warm drive.

Also note that a few of the warm buckets are quite large, so e.g. drwx--x--x 3 root root 12288 Jan 23 00:00 db_1326966040_1319190041_165 represents more than half of the storage used in hot/warm and has been sitting there for 2+ weeks.

These things together (large variations in individual bucket sizes and weird rolling rules) are making it hard to predict how much local storage is utilized -- it's been creeping up from 60% of disk 2 weeks ago to presently 80%.

cat ./etc/system/local/indexes.conf
# note - known bug, do not put comments at end of active config lines

################################################################################
# index definitions
################################################################################

[main]
homePath       = $SPLUNK_DB/defaultdb/db
coldPath       = /nfs_mount_path/splunk/colddb
maxWarmDBCount = 25

HOT/WARM $ ls -ltd hot_* db_*

drwx--x--x 3 root root 12288 Feb 8 12:45 hot_v1_253
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_quar_v1_258
drwx--x--x 3 root root 12288 Feb 8 12:45 hot_v1_235
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_259
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_260
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_268
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_271
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_273
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_274
drwx--x--x 3 root root 4096 Feb 8 12:45 hot_v1_275
-rw------- 1 root root 0 Feb 8 12:23 hot_v1_235.lock
drwx--x--x 3 root root 4096 Feb 5 00:01 db_1319331752_1311556906_239
drwx--x--x 3 root root 4096 Feb 5 00:00 db_1320410401_1319339003_257
drwx--x--x 3 root root 4096 Jan 31 00:01 db_1319058648_1318332360_225
drwx--x--x 3 root root 12288 Jan 30 13:59 db_1327965874_1320411621_226
drwx--x--x 3 root root 12288 Jan 30 11:27 db_1327734209_1320410736_203
drwx--x--x 3 root root 4096 Jan 27 10:43 db_1318332360_1318332360_223
drwx--x--x 3 root root 4096 Jan 26 12:42 db_1318332360_1318332360_221
drwx--x--x 3 root root 4096 Jan 26 00:00 db_1318332360_1318332360_217
drwx--x--x 3 root root 12288 Jan 23 00:00 db_1326966040_1319190041_165
drwx--x--x 3 root root 4096 Jan 22 00:00 db_1318944946_1314187526_200
drwx--x--x 3 root root 4096 Jan 15 00:00 db_1317630478_1309936946_181
drwx--x--x 3 root root 4096 Jan 15 00:00 db_1318332360_1318332360_193
drwx--x--x 3 root root 4096 Jan 12 18:37 db_1317755181_1317753460_190
drwx--x--x 3 root root 4096 Jan 12 00:00 db_1317839457_1317752327_187
drwx--x--x 3 root root 4096 Jan 9 00:00 db_1318332360_1314188057_176
drwx--x--x 3 root root 4096 Jan 1 00:00 db_1320411770_1317218970_168
drwx--x--x 3 root root 4096 Dec 29 00:00 db_1324992170_1317216171_150
drwx--x--x 3 root root 12288 Dec 28 11:59 db_1325100656_1324992171_164
drwx--x--x 3 root root 12288 Dec 22 12:23 db_1324591875_1324562133_145
drwx--x--x 3 root root 12288 Dec 22 12:23 db_1324592143_1324573951_148
drwx--x--x 3 root root 12288 Dec 22 11:52 db_1324589857_1320410738_131
drwx--x--x 3 root root 12288 Dec 22 11:52 db_1324590483_1324572091_143
drwx--x--x 3 root root 12288 Dec 20 13:41 db_1324406345_1324405289_130
drwx--x--x 3 root root 4096 Dec 20 13:41 db_1324421476_1324358838_128
drwx--x--x 3 root root 4096 Dec 20 12:27 db_1324400968_1324357038_127

COLD

$ ls -lt

drwx--x--x 3 root sw 321 Feb 8 11:35 db_1261018288_1254902958_267
drwx--x--x 3 root sw 321 Feb 8 11:35 db_1272887488_1265647707_269
drwx--x--x 3 root sw 321 Feb 8 11:35 db_1299670749_1296654669_272
drwx--x--x 3 root sw 321 Feb 8 11:34 db_1309936932_1309936932_270
drwx--x--x 3 root sw 321 Feb 7 23:59 db_1268038207_1260963083_264
drwx--x--x 3 root sw 321 Feb 7 23:59 db_1289227711_1284385042_265
drwx--x--x 3 root sw 321 Feb 7 23:59 db_1299687158_1294127838_262
drwx--x--x 3 root sw 321 Feb 6 14:23 db_1278925604_1272887478_266
drwx--x--x 3 root sw 320 Feb 6 10:34 db_1257953943_1257953943_263
drwx--x--x 3 root sw 321 Feb 6 10:34 db_1284384925_1278925589_261
drwx--x--x 3 root sw 364 Feb 5 23:55 db_1304366294_1296654671_238
drwx--x--x 3 root sw 321 Feb 4 23:56 db_1257954177_1247585038_245
drwx--x--x 3 root sw 363 Feb 4 23:56 db_1278260593_1278260000_256
drwx--x--x 3 root sw 321 Feb 4 23:56 db_1286365262_1278925548_241
drwx--x--x 3 root sw 364 Feb 4 23:56 db_1294135872_1289211918_242
drwx--x--x 3 root sw 363 Feb 4 23:56 db_1311354187_1304434263_255
drwx--x--x 3 root sw 321 Feb 4 23:42 db_1317630085_1309936972_212
drwx--x--x 3 root sw 321 Feb 4 23:40 db_1317630081_1309937121_229
drwx--x--x 3 root sw 321 Feb 3 23:44 db_1247585102_1155734602_236
drwx--x--x 3 root sw 321 Feb 3 23:44 db_1268038236_1260963083_246
drwx--x--x 3 root sw 321 Feb 2 23:58 db_1275641578_1275641578_254
drwx--x--x 3 root sw 321 Feb 2 16:01 db_1257954195_1257954195_251
drwx--x--x 3 root sw 321 Feb 2 16:01 db_1307524881_1307520654_252
drwx--x--x 3 root sw 321 Feb 1 23:58 db_1270714365_1270714365_250
drwx--x--x 3 root sw 321 Feb 1 15:44 db_1257954195_1257954195_248
drwx--x--x 3 root sw 321 Feb 1 15:44 db_1275641595_1275641559_247
drwx--x--x 3 root sw 321 Feb 1 15:44 db_1307520923_1307520923_249
drwx--x--x 3 root sw 321 Feb 1 15:44 db_1309937136_1307520589_244
drwx--x--x 3 root sw 321 Jan 31 19:49 db_1272887603_1262795698_237
....
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1247584874_1244723499_26
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1244723646_1165832896_64
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1244723499_1244723499_50
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1244723499_1155734758_95
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1244723499_1244723499_41
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1236752810_1152078946_44
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1239695295_1165832896_24
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1236752810_1165832896_84
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1236752810_1165832896_33
drwx--x--x 3 root sw   321 Dec 22 11:40 db_1205971033_1205744465_75

[jbsplunk]

Please keep in mind that we aren't using the mtime of the file, which is what it sounds like you're assuming based on the description you've provided. We are using the epoch time of the bucket to determine what is oldest.

Notice the format of the warm buckets is in db_epoch_epoch_bucketid. The first epoch time is the earliest event in the bucket, and the second epoch time is the earliest event in a bucket. The behavior I would expect to see is that once you've reached 25 warm buckets, the oldest bucket based on the epoch times contained in the buckets would be moved to cold.

To see why a bucket is being moved, you can look at the BucketMover component of splunkd.log, which will tell you when a bucket is moved and why it was moved. I would expect the next bucket which would be moved is going to be this one:

drwx--x--x 3 root root 4096 Feb 5 00:00 db_1320410401_1319339003_257

Hope this helps explain the behavior.

You might find this link, which explains the retention settings in detail, to be helpful:

http://docs.splunk.com/Documentation/Splunk/latest/admin/HowSplunkstoresindexes

[forrest_whitche]

Ok I see what you're saying -- "The first epoch time is the earliest event in the bucket, and the second epoch time is the earliest event in a bucket"

(I assume you actually mean - the second epoch time is the latest event in a bucket)

I still can't see how it's not working 'as designed'. The largest warm bucket spans Oct22'11 to Jan19'12 and many other buckets have been moved to cold while that one sits in warm. See also all the buckets dated ca Dec 22 (with epoch times to match) that are sitting there.

I'm going to down splunk and move these manually later today.

[forrest_whitche]

I would expect the next bucket which would be moved is going to be this one:
drwx--x--x 3 root root 4096 Feb 5 00:00 db_1320410401_1319339003_257

Nope, what moved next was 286 then 273, 181, 276, 277.

What I do see is that some of my highest-volume logs are contained in the 165 bucket that's stubbornly staying put. -- makes sense.

I will look over splunkd.log and see what I see, thanks!