How to troubleshoot "ERROR BucketMover - freeze failed"?

[ffr03]

Hi guys,

I am getting an error message in Splunk Internal Errors and Messages

_raw    _time   host    index   linecount   log_level   source  sourcetype  splunk_server
05-09-2016 17:00:47.686 +0100 ERROR BucketMover - freeze failed: failed for bkt='D:\SplunkData\colddb\db_1436786438_1436782239_1034'failed to rename src='D:\SplunkData\colddb\db_1436786438_1436782239_1034' to dst='D:\SplunkData\colddb\inflight-db_1436786438_1436782239_1034' (reason='The operation completed successfully.'); result='Rename failed in 15 attempt(s) made between Mon May 09 17:00:31 2016 and Mon May 09 17:00:46 2016 [status code: 5]'    2016-05-09 17:00:47 XXXX    _internal   1   ERROR   D:\Splunk\var\log\splunk\splunkd.log    splunkd XXXX
05-09-2016 17:00:47.373 +0100 ERROR BucketMover - freeze failed: failed for bkt='D:\SplunkData\colddb\db_1436781698_1436777439_1007'failed to rename src='D:\SplunkData\colddb\db_1436781698_1436777439_1007' to dst='D:\SplunkData\colddb\inflight-db_1436781698_1436777439_1007' (reason='The operation completed successfully.'); result='Rename failed in 15 attempt(s) made between Mon May 09 17:00:31 2016 and Mon May 09 17:00:45 2016 [status code: 5]'    2016-05-09 17:00:47 XXXX    _internal   1   ERROR   D:\Splunk\var\log\splunk\splunkd.log    splunkd XXXX

Can anyone help me trace this issue?

[lycollicott]

Did you ever get this fixed? It is very common on Windows based indexers and there is no fix from Splunk, but you can work around it.

I used to see this sort of thing happen all the time, but I have basically eliminated it now - more on that later.

On Windows Splunk creates folders with Full control on This folder only and that is where the problem seems to stem from. For example:

The first ID listed there is our service account that Splunk runs as and the second line is the local Administrators group which the service account belongs to.

Whenever I encounter a BucketMover error accessing an inflight* folder I change the permissions on the parent folder. So, for your error on D:\SplunkData\colddb\inflight-db_1436786438_1436782239_1034 I would change the permissions on D:\SplunkData\colddb from This folder only to This folder, subfolders and files.

That is the workaround, but how did I eliminate these errors? Well I can't be certain what caused the very first one, but I made the problem worse when I began to investigate different folders to compare settings. I was a local administrator, but I was prompted with the message You don't currently have permission to access this folder. Click Continue to permanently get access to this folder. It took me a very long time to figure it out, but what was happening was that when I granted myself access like this I became the only userid with full control of that folder. As soon I realized that and I stopped doing it then the issue stopped.

So, if you ever need to browse your buckets with Windows Explorer and are prompted to grant yourself access just remember to make that This folder, subfolders and files change.