Deployment Architecture

splunk 6.6.4 search head >> indexer peer down!!!

kedjjang
Explorer

[ Environment ]
SPLUNK 6.6.4 Search Head Cluster 3 EA
SPLUNK 6.6.4 Indexer Cluster 4EA

Server specifications are very good.

[distsearch.conf] - searcher
connectionTimeout , sendTimeout , receiveTimeout
[limits.conf] - searcher , indexer
base_max_searches ,max_rt_search_multiplier , max_searches_per_cpu

We have set this option to a value that is better than the default value.

If there are many searches on the search server, the indexer is not actually dead, but the error is still happening.

[error msg]
Unable to distribute to peer named * at uri=* using the uri-scheme=https because peer has status="Down". Please verify uri-scheme, connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more informati

[internal log]
ERROR StreamedSearch - sid=remote_adminadmin___RMD561dff7a2ba7b014f_1525229108.3657_03943F0E-D90E-4C81-BA9B-D2F89B841083, Broken pipe

WARN HttpListener - Socket error from ... while accessing /services/streams/search: Broken pipe

Do you know the cause of the symptom and the solution?

0 Karma

xpac
SplunkTrust
SplunkTrust

I'd guess that the indexer is under such heavy load that the connection times out.

Please show us your actual "better values" for the settings above, and also take a close look at your monitoring console about Ressource usage etc.

0 Karma

kedjjang
Explorer

Thank you for your comment.

distsearch.conf has been doubled from its default value.

limits.conf The default has been increased by a factor of two or three, and the resources have not changed much.

However, the same symptoms.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...