Search Peer status keeps changing every minute. Why?

[DerekB]

The search head server is giving various errors communicating with search peers/indexers. It keeps bouncing various error notifications every other minute from search peer to search peer.

• Unable to distribute to peer named SH1 at uri https://SH1:8089 because peer has status = "Down". Unable to get bundle list -Unable to distribute to peer named SH2 at uri https://SH2:8089 because peer has status = "Down". Unable to get bundle list

Search Peer status keeps changing from minute to minute

SH1:8089 sh1 Up Successful
SH3:8089 sh3 Up Successful
SH4:8089 SH4 Down Initial
SH5:8089 sh5 Up Successful

sh2:8089 SH2 Up Successful

SH1:8089 SH1:8089 Down Initial
SH2:8089 sh2 Down Initial
SH3:8089 SH3 Up In progress
SH1:8089 SH1 Up In progress
SH2:8089 SH3 Authentication Failed Initial
sh4:8089 SH4 Up In progress

What the heck is going on?

[DerekB · Accepted solution]

There was a rogue nic card that came online at some point after the upgrade a couple of weeks ago and picked up a secondary ip address and became the preferred path. However this rogue address was not routable across the network and hence the intermittent connections to the indexers. We could not detect this because the main interface was still working fine when connecting to the server remotely. Thanks for mentioning physical location, that led me to logon to the server again and check the network interfaces and route table and found the problem.