Getting low on warm space for my buckets, so I changed the maxHotSpanSecs to 6100000 or ~70 days. After restarting the indexer, I'm getting the error messages in the title. Can someone assist w/ this?
Did you restart Splunk as the wrong user? Typically only applies to linux distros.
If so you'll need to stop Splunk and then recursively change the owner to fix file permission issues:
chown -Rf correctUser:correctGroup /path/to/splunk
Then switch to the correct user and start Splunk.
It's running as the right user, and there are new buckets from the indexes (with the correct permissions), but I'm still getting the error message. I was also expecting more buckets in the cold directories/partition, after changing the setting and restarting.
Hot buckets (typically only 3) roll to warm buckets (typically 200+) before they role to cold.
Does the fact that I changed the maxHotSpanSecs speed up the rolling process (instead of 1 roll of three buckets per day, more buckets roll per day)? If not, how do I do that? I'd like to clear up some space on my warm partition of data we don't search that often.
Bucketing and retirement policies are tricky. We kinda got off subject though. I have to read the docs every time I mess with indexes.
http://docs.splunk.com/Documentation/Splunk/6.0.2/Indexer/Setaretirementandarchivingpolicy
The bucket roll will occur on a restart but it can take some time. It also needs to be triggered by the settings. http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes
In short... All events in the bucket need to match the condition prior to the roll.
There is also a command that will force bucket rolls.
To speed up the roll, given the fact that all events must match the conditions, we typically reduce the max data size and frozen time period:
The maxTotalDataSizeMB and frozenTimePeriodInSecs attributes in indexes.conf help determine when buckets roll from cold to frozen.
I'm looking to roll from warm to cold, so I'd set maxWarmDBCount to less than 300 I'd imagine. Don't see why I changed maxHotSpanSecs now that I think about it. Does that sound right for what I'm trying to do?
Also, can you think of why I'm getting the error regarding recursive copy?
Thanks very much for your help, btw.
Agreed, the maxHotSpanSecs isn't as much help as the maxTotalDataSizeMB and maxDataSize. Sorry i was on mobile, but now home... this is the end all resource you need to thoroughly read & understand: http://docs.splunk.com/Documentation/Splunk/6.0.1/admin/Indexesconf
The recursive copy error was most likely due to file permissions. Perhaps splunk ran as root once upon a time... made some hot/warm buckets owned by root, and now your new setting is trying to move those buckets. You should be able to find more issues "around/near" that error message if you look in index=_internal component=bucketmover
or just index=_internal
. Its telling you it cant find the file to copy, or cant find the destination to copy to. So either coldDBPath / WarmDBPath is not available... or you dont have permission to access those paths, or you dont have permission to access the bucket its trying to roll (by you I mean the splunkd user).
03-14-2016 14:26:59.302 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/application/db/db_1354776505_1354743198_475' to dst='/mnt/splunk/cold/appl
ication/colddb/inflight-db_1354776505_1354743198_475' failed (reason='No such file or directory')
03-14-2016 14:26:59.333 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/security/db/db_1354759131_1354748274_3081' to dst='/mnt/splunk/cold/securi
ty/colddb/inflight-db_1354759131_1354748274_3081' failed (reason='Permission denied')
03-14-2016 14:26:59.999 -0400 ERROR BucketMover - aborting move because recursive copy from src='/mnt/splunk/hot_warm/application/db/db_1354743197_1354664613_474' to dst='/mnt/splunk/cold/appl
ication/colddb/inflight-db_1354743197_1354664613_474' failed (reason='No such file or directory')
/mnt/splunk/cold/application/colddb/inflight-db_1354743197_1354664613_474:
total 304624
drwx--x--x 3 splunk splunk 4096 2016-03-14 14:29 .
drwx------ 721 splunk splunk 53248 2016-03-14 14:28 ..
-rw------- 1 splunk splunk 17888225 2016-03-14 14:29 1354742336-1354738684-6468197426103594966.tsidx
-rw------- 1 splunk splunk 293102760 2016-03-14 14:28 1354743197-1354664613-2556498674997796822.tsidx
/mnt/splunk/cold/application/colddb/inflight-db_1354776505_1354743198_475:
total 86908
drwx--x--x 3 splunk splunk 4096 2016-03-14 14:28 .
drwx------ 721 splunk splunk 53248 2016-03-14 14:28 ..
-rw------- 1 splunk splunk 77457453 2016-03-14 14:28 1354773028-1354743198-1990108998233234267.tsidx
-rw------- 1 splunk splunk 11048518 2016-03-14 14:28 1354775364-1354753985-1692011677748308117.tsidx
I think we get an error, and then the copy continues anyway. That's what this looks like anyway. Also, do I want to adjust maxTotalDataSizeMB, isn't that for all the data in the index?maxDataSize is for the hot buckets I think. Should I edit maxWarmDBCount to roll warm to cold? That says to roll at a certain number, but there's less control over the amount of time the data spends in warm buckets, right?
Yep it's a permission denied error. You should try the recursive chown. Bucket mover errors can quickly stack up. maxDataSize and the number of buckets hot, and warm are the ones you're most interested in.
As well as frozenTimePeriodInSecs if you're wanting to roll to frozen/delete