Problem replicating config (bundle) to search peer ' xx.x.xx.xx:8089 ', Unknown write error.

[danthebiman]

i have a relative simple setup. One instance is an indexer, another is search head and heavy forwarder. All seems fine, except when i added the indexer as a search peer. That's when I see the message saying "Problem replicating config (bundle) to search peer ' xx.x.xx.xx:8089 ', Unknown write error." on the search head/heavy forwarder. But no message on the indexers WebUI. Here are the splunkd.log I pulled from both boxes.

Error found on the search head/heavy forwarder

08-12-2019 23:01:29.617 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:29.619 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:29.629 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:29.630 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:29.631 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:30.525 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:30.526 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:38.729 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:38.731 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:54.815 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:54.816 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:54.817 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:01:56.285 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:01:56.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:01:56.287 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:03:41.074 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50062 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:03:41.075 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:03:41.075 -0400 WARN HttpListener - Socket error from [My Laptop]:50061 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:05:09.870 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:05:09.872 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:05:31.286 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=Hardware_DuplicateMacsToEliminate.csv will attempt to use implicit filename.
08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=OU.csv will attempt to use implicit filename.
08-12-2019 23:05:31.288 -0400 WARN CsvDataProvider - Unable to find filename property for lookup=TypeLookupObjectRoleContainer.csv will attempt to use implicit filename.
08-12-2019 23:13:37.607 -0400 INFO KeyManagerLocalhost - Sending public key to search peer: https://[INDEXER IP]:8089
08-12-2019 23:13:37.612 -0400 INFO KeyManagerLocalhost - Sent public key to search peer: https://[INDEXER IP]:8089
08-12-2019 23:13:37.619 -0400 INFO ServerConfig - Using REMOTE_SERVER_NAME=[SH/HF Host Machine]
08-12-2019 23:13:37.623 -0400 INFO ServerRoles - Declared role=search_head.
08-12-2019 23:13:43.523 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:13:43.523 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:17:00.996 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:17:01.291 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:17:05.469 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:17:05.470 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:18:09.673 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:18:09.673 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:21:14.523 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:21:14.526 -0400 WARN DistributedPeerManager - Unable to distribute to peer named [INDEXER Host Machine] at uri https://[INDEXER IP]:8089 because replication was unsuccessful. ReplicationStatus: Failed - Failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE. Verify connectivity to the search peer, that the search peer is up, and that an adequate level of system resources are available. See the Troubleshooting Manual for more information.
08-12-2019 23:21:18.741 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:21:18.741 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.
08-12-2019 23:22:22.986 -0400 INFO NetUtils - SSL_write failed. Connection reset by peer
08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Unexpected problem while uploading bundle: Unknown write error
08-12-2019 23:22:22.986 -0400 ERROR DistributedBundleReplicationManager - Problem replicating config (bundle) to search peer ' [INDEXER IP]:8089 ', Unknown write error.
08-12-2019 23:22:22.987 -0400 ERROR DistributedBundleReplicationManager - Unable to upload bundle to peer named [INDEXER Host Machine] with uri=https://[INDEXER IP]:8089.

Error found on the indexer

08-12-2019 23:00:09.326 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/log/introspection/resource_usage.log'.
08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50080 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.243 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:29.243 -0400 WARN HttpListener - Socket error from [My Laptop]:50081 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.731 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:29.731 -0400 WARN HttpListener - Socket error from [My Laptop]:50083 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:29.732 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:29.732 -0400 WARN HttpListener - Socket error from [My Laptop]:50082 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:32.469 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:32.469 -0400 WARN HttpListener - Socket error from [My Laptop]:50084 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:32.475 -0400 WARN SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client key exchange A', alert_description='certificate unknown'.
08-12-2019 23:13:32.475 -0400 WARN HttpListener - Socket error from [My Laptop]:50085 while idling: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
08-12-2019 23:13:37.610 -0400 INFO KeyManagerSearchPeers - Updating public key for search peer: [SH/HF Host Machine]
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished reading public key for peer: /opt/splunk/etc/auth/distServerKeys/[SH/HF Host Machine]/trusted.pem
08-12-2019 23:13:37.611 -0400 INFO KeyManagerSearchPeers - Finished updating public key for search peer: [SH/HF Host Machine]
08-12-2019 23:13:43.530 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43164 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:13:43.530 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43164 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:14:12.583 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:17:05.469 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43266 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:17:05.470 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43266 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:17:18.584 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:18:09.678 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43280 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:18:09.678 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43280 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:18:20.584 -0400 WARN DistributedMetrics - Invalid bundle status
08-12-2019 23:21:18.749 -0400 ERROR HttpListener - Exception while processing request from [SH/HF IP]:43324 for /services/receivers/bundle/[SH/HF Host Machine]: Connection closed by peer
08-12-2019 23:21:18.749 -0400 WARN HttpListener - Socket error from [SH/HF IP]:43324 while accessing /services/receivers/bundle/[SH/HF Host Machine]: Broken pipe
08-12-2019 23:21:26.584 -0400 WARN DistributedMetrics - Invalid bundle status

Any Help is much appreciated.