Deployment Architecture

After upgrading my Indexer Cluster to 6.5.1, why is the replication status "pending"?

rafamss
Contributor

Hi dears,

I have * 21 indexers * in my Splunk environment running in index cluster mode. After upgrading the whole site from version 6.3.1 to version 6.5.1, I have the problem with ** replication data **. Invariably some machines fall Down and stay with Pending Status, and then these machines come back Up with Status Searchable. This process, so to speak, occurs several times.

Could someone tell me what that could be? I put two attachments to help.

dxu_splunk
Splunk Employee
Splunk Employee

You'll need to keep indexers of the same cluster within one minor version of each other.

So, 6.4.X indexers will be okay with 6.5.x indexers, however 6.3.x and 6.5.x indexers will not be guaranteed (and indeed 6.3.x <-> 6.5.x+ replication is intentionally broken)

jcrabb_splunk
Splunk Employee
Splunk Employee

You may want to consider going to 6.5.2 as there are two bugs that can impact a busy environment.

6.5.2 Release Notes

  • SPL-134427, SPL-133450: 6.5+ splunk does full bundle replication everytime - slowing down the system
  • SPL-131398, SPL-132804, SPL-132805, SPL-132807, SPL-132890: Search head cluster contention on Linux due to poor hashing inside OpenSSL's error container.

It is possible that either of these are causing some contention which results in a peer timing out while trying to communicate with the Cluster Master. The result of that scenario is a peer with a "Status" that is fluctuating.

If that doesn't correct the issue and you have more than 20-30K buckets per indexer, some timings may need to be adjusted but I would highly encourage you to upgrade first.

Jacob
Sr. Technical Support Engineer
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...