Deployment Architecture

Why are we unable to remove excess buckets in a multisite indexer clustering environment?

rbal_splunk
Splunk Employee
Splunk Employee

We are in Multi-site clustered environment with Site RF and SF= 3 and for different indexes, we see a number of buckets listed as excess buckets. Due to some reason, we are unable to remove these buckets

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Run the search below to get the list of extra buckets - you will run this on the cluster master. Note I have set “replication_count>3” which means it will list only buckets that have a number of copies above 3.

| rest /services/cluster/master/buckets filter=replication_count>3| rex field=title "^(?<repl_index>[^\~]+)"
   | search repl_index="*" standalone=0 frozen=0
   | rename title AS bucketID
   | fields bucketID peers.*.search_state  *site*
   | untable bucketID siteState value
   | rex field=siteState "peers\.(?<peerGUID>[^\.]*?)\.(?<siteState>search_state)"
   | rex field=siteState "(?<siteState>primaries_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>rep_count_by_site)\.(?<site>\S+)"
   | rex field=siteState "(?<siteState>search_count_by_site)\.(?<site>\S+)"
   | eval peerGUID=if(siteState=="primaries_by_site", value, peerGUID)
   | eval site=if(siteState=="origin_site", value, site)
   | eval value=if(siteState=="search_count_by_site", site + ":" + value, value)
   | eval value=if(siteState=="rep_count_by_site", site + ":" + value, value)
   | join type=outer peerGUID [ rest /services/cluster/master/peers
                          | fields active_* host* label title status site
                          | eval PeerName= site + ":" + label + ":" + host_port_pair
                          | rename title AS peerGUID
                          | rename site AS peerSite
                          | table peerGUID PeerName peerSite ]
   | eval site=if(siteState=="search_state", peerSite, site)
   | eval value=if(siteState=="primaries_by_site", PeerName + ":For_" + site, value)
   | eval value=if(siteState=="search_state", PeerName + ":" + value, value)
   | fields - PeerName peerGUID peerSite    | chart values(value) over bucketID by siteState 

For the list of the Buckets listed - check the Bucket REST end point using URL

https://:/services/cluster/master/buckets/

like.....

https://:/services/cluster/master/buckets/_audit~1~ABE7B836-1BD4-4EBD-8F2F-740DAE1DB9F4

check these buckets' REST end point for attribute "constrain_to_origin_site" . If this attribute has a value of '1' - it means that bucket was created before enabling multisite. These buckets won't be removed for excess copies.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...