Getting Data In

Handle lost buckets

asnegina
New Member

I have splunk cluster with sf=2, rf=2 (they are met).
It was maintained by another contractor, so I have no ideas about what caused the issue.
First, on master node for some indexes I see data copies marked grey and data copies number is 170/173, etc... same for serachable and replicated copies. And no bucket fixup tasks are running.
I guess this means that some buckets do not exist on both indexers, am I right?

Second, I ran dbinspect command for this index, it returned 173 results. tsidxState is "full" for every bucket.

How do I find problematic buckets and delete them to make Indexer clustering page green again?

UPD. I uploaded photo
alt text

0 Karma

ivanreis
Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

asnegina
New Member

Thank you for your answer. As I mentioned before I have no buckets in fixup state when I click on gray ones. Now it looks like a visual bug but it still persists after server restart.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Have you tried to restart splunk on Cluster Master? Sometimes I have seen that CM flags indexes with grey color but on top of the screen it displays Search Factor and Replication Factor are met & after few minutes every indexes were green.

asnegina
New Member

I tried, but it does not help. All blocks for particular indexes are constantly grey. You can check the photo I added to my question (can't make screenshot, sorry)

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

If you click on that grey icon for one of the index, it will take you to Bucket Status page, as you mentioned you don't have any bucket fixup running but can you please check any fixup tasks - pending ?

0 Karma

asnegina
New Member

I was incorrect, actually I have no fixup tasks at all (pending or in progress), and no excess buckets btw

0 Karma

ivanreis
Builder

try to fix the bucket editing the gray ones :

For buckets that have been stuck in fixup for long periods of time, you can take remedial action.

Click Action for the bucket that you want to manage.
Select one of the available actions:
View bucket details
Roll
Resync
Delete Copy
A pop-up window appears to guide you through the selected action.
Use the following sequence when performing actions on anomalous bucket.

View bucket details
Roll
Resync
Delete Copy
Only perform the next action if the previous one does not resolve the issue.

For further information, check this link -> https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Anomalousbuckets#Take_action_on_an_anomal...

If it did not work, please submit a case to splunk support and generate a diag file to attach to the case running ./splunk diag

0 Karma

yannK
Splunk Employee
Splunk Employee

Ivan, please post your method as an answer instead of a comment. It will help mark the question as answered.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...