Deployment Architecture

Multisite Indexer Cluster Inconsistent Buckets: How to delete buckets and their metadata to no longer be listed in the cluster master REST endpoint?

tluluma3
New Member

Every time when I do a splunk rolling-restart cluster-Peers I have some buckets that are listed in the REST Endpoint, but they are not on the filesystem on the particular Indexer.
I can find these buckets with:

index=_internal host="CLUSTERMASTER" sourcetype=splunkd from=Unsearchable | stats values(peer_name) by bid

I then can delete the buckets with splunk _internal call /services/cluster/master/buckets/BID/remove_all -method POST, but they reappear after the next rolling restart.

Is there a way to delete the metadata from these buckets for the Cluster master to "forget" them forever?

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

seems like you want to delete a bucket entirely.

if remove_all doesn't work (it might not work if the bucket is in some transitory state, ie PendingSearchable), you can try freezing the bucket on the indexers.

curl -k -u USER:PASS https://indexer:mgmt_port/services/data/indexes/INDEX/freeze-buckets -d bucket_ids=117_22220097-5E3F-4D26-B301-ECE3C4CD2222 -X POST

bucket_ids is a list of buckets by ID_GUID. this should remove the bucket on that specific indexer - just repeat for all the other copies of the bucket

0 Karma

dxu_splunk
Splunk Employee
Splunk Employee

can you post the logs for a specific bucket ?

0 Karma

tluluma3
New Member
10-15-2015 13:48:43.465 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:42:08.936 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:46:07.692 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:39:28.442 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:19.039 +0200 INFO CMPeer - peer=DF9EF4CB-14CE-4F5B-8BA0-9D687671B4B6 peer_name=oltapssp-slin02.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
10-15-2015 13:36:00.952 +0200 INFO CMPeer - peer=7EBE2435-3FB3-4CE2-85E3-35931CD26293 peer_name=zhhapssp-slin05.it.bwns.ch bid=_internal~11942~14EE1C69-1AED-4A51-8A8E-0FDCBE2C16FC transitioning from=Unsearchable to=PendingSearchable oldmask=0x0 newmask=0x0 reason="fixup searchable count"
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...