Deployment Architecture

Cluster bundle stuck in validation mode?

rbal_splunk
Splunk Employee
Splunk Employee

We have 40 node Indexer cluster setup on Splunk version 6.6.
We push a bundle and only 3 of the 32 indexers have validated the package. We are stuck and cannot push out any updates to indexers.

alt text

0 Karma
1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

Important point to consider is when the cluster master sends the bundle to each of the peer, this triggers  bundle validation on the peers. After the bundle validation is complete, the peers send a validation response to the cluster master.

The cluster master waits for a response from all the cluster peers before it can proceed to bundle push state.
Even if one peer does not send the response, the cluster master waits for that peer indefinitely. That's why we have cancel bundle push in 7.0.

To avoid such situation, before pushing the bundle , please confirm that "Splunk show cluster-bundle-status" shows "cluster_status" as None.

Now to Debug the current situation, it's needed to enable debug for component CMSlave on Cluster peer, and that will show you something like below , and it will show you which is active bundle on cluster peer.

CMBundleMgr - setting active bundle

Normally to get out of such situation you need to restart the cluster peer . When the cluster peer is restarted it reset the state of the Cluster peer.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...