Every time when I do a splunk rolling-restart cluster-Peers
I have some buckets that are listed in the REST Endpoint, but they are not on the filesystem on the particular Indexer.
I can find these buckets with:
index=_internal host="CLUSTERMASTER" sourcetype=splunkd from=Unsearchable | stats values(peer_name) by bid
I then can delete the buckets with splunk _internal call /services/cluster/master/buckets/BID/remove_all -method POST
, but they reappear after the next rolling restart.
Is there a way to delete the metadata from these buckets for the Cluster master to "forget" them forever?
seems like you want to delete a bucket entirely.
if remove_all doesn't work (it might not work if the bucket is in some transitory state, ie PendingSearchable), you can try freezing the bucket on the indexers.
curl -k -u USER:PASS https://indexer:mgmt_port/services/data/indexes/INDEX/freeze-buckets -d bucket_ids=117_22220097-5E3F-4D26-B301-ECE3C4CD2222 -X POST
bucket_ids is a list of buckets by ID_GUID. this should remove the bucket on that specific indexer - just repeat for all the other copies of the bucket
can you post the logs for a specific bucket ?
10-15-2015 13:48:43.465 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:42:08.936 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:46:07.692 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:39:28.442 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:19.039 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:00.952 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"