Splunk Search

Bundle reload is in progress. Waiting for all peers to return the status.

phanichintha
Path Finder

My splunk environment is:
1 Search Head
1 Deployment Server (Master Node)
2 Indexers (Cluster)

I tried to implement retention policy to delete more than 365days old data in indexer, so i implement the sample script for one of the index(_internal).
So i first changed the indexes.conf under D:\Splunk\etc\master-apps_cluster\local in Master Node.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

After same changed the indexes.conf under D:\Splunk\etc\slave-apps_cluster\local in indexer.
**Indexes.conf:
[_internal]
coldToFrozenDir = $SPLUNK_DB/_internal/frozendb
coldPath = $SPLUNK_DB/_internal/colddb
homePath = $SPLUNK_DB/_internal/db
thawedPath = $SPLUNK_DB/_internal/thaweddb

frozen time is 365 days

frozenTimePeriodInSecs = 31536000
maxHotIdleSecs = 3600
repFactor = auto*****

For doing changes, in master node GUI under settings > indexer cluster > push button.
After doing this from more than 3 hours its still (Bundle reload is in progress. Waiting for all peers to return the status.)
what is the time taken by doing this. Please any one help on priority.

0 Karma

codebuilder
Influencer

The reason you are having issues is that you manually edited indexes.conf on the indexers. This should never be done when you have clustered indexers controlled by a master node.

When you want to update/change indexes.conf on your cluster, you only modify the file on the master at /opt/splunk/etc/master-apps/_cluster/local/indexes.conf (linux). When you push the new bundle to the indexers from the master, the updated indexes.conf is written to /opt/splunk/etc/slave-apps/_cluster/local/indexes.conf on the peers (indexers).

The problem you are seeing is that the master cannot verify that the peers have the correct previous bundle because you modified them manually, and therefore will not push out the new changes.

Revert the manual changes you made on the peers/indexers and re-push the updated version from the master.

----
An upvote would be appreciated and Accept Solution if it helps!

codebuilder
Influencer

Also, the value you posted for repFactor is not valid..
You provided "repFactor = auto***"

The asterisks are not needed nor correct.
The value should be simply "repFactor = auto"

Worth noting as well, a two node indexer cluster is not good practice (though not the cause of your current issues). With only two nodes the cluster can never reach a quorum, so you're introducing a split-brain scenario.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...