Deployment Architecture

splunk 6.6.4 search head >> indexer peer down!!!

kedjjang
Explorer

[ Environment ]
SPLUNK 6.6.4 Search Head Cluster 3 EA
SPLUNK 6.6.4 Indexer Cluster 4EA

Server specifications are very good.

[distsearch.conf] - searcher
connectionTimeout , sendTimeout , receiveTimeout
[limits.conf] - searcher , indexer
base_max_searches ,max_rt_search_multiplier , max_searches_per_cpu

We have set this option to a value that is better than the default value.

If there are many searches on the search server, the indexer is not actually dead, but the error is still happening.

[error msg]
Unable to distribute to peer named * at uri=* using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more informati

[internal log]
ERROR StreamedSearch - sid=remote_adminadmin___RMD561dff7a2ba7b014f_1525229108.3657_03943F0E-D90E-4C81-BA9B-D2F89B841083, Broken pipe

WARN HttpListener - Socket error from ... while accessing /services/streams/search: Broken pipe

Do you know the cause of the symptom and the solution?

0 Karma

xpac
SplunkTrust
SplunkTrust

I'd guess that the indexer is under such heavy load that the connection times out.

Please show us your actual "better values" for the settings above, and also take a close look at your monitoring console about Ressource usage etc.

0 Karma

kedjjang
Explorer

Thank you for your comment.

distsearch.conf has been doubled from its default value.

limits.conf The default has been increased by a factor of two or three, and the resources have not changed much.

However, the same symptoms.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...