Splunk Search

"Unable to distribute to peer named...because peer has status = "Down""?

vrmandadi
Builder

What does this error mean?

Unable to distribute to peer named foobar237.xxx.com:8089 at uri https://foobar237.xxx.com:8089 because peer has status = "Down". 
0 Karma

spinnamshetty
New Member

what i found in my case is,
when the search head went down , i found out there are some "REAL-TIME" searches were running by other users.
and for confirmation i have checked DMC on my search head ,and i got the same thing at what time "SEARCH" process taken more RAM and CPU ,
then i came to one conclusion that , because of some weird searches, my SH went down
hope this helps

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vrmandadi,
we have the same problem caused by an high use of CPUs: on indexers we have 12CPUs but sometimes we have at the same time more than 20 scheduled searches so there's a queue and after some time there's a disconnection for timeout (peer has status = "Down". ).
You can check this using Monitoring Console (Resource usage: instance, 90th Percentile CPU Usage by Process Class).

Splunk Support suggested to optimize searches, give more CPUs to the system and don't use higher timeout values.

We're working to do this, I'll inform you!

Bye.
Giuseppe

gcusello
SplunkTrust
SplunkTrust

Hi Hi vrmandadi,
we solved the problem optimizing searches: there was a very heavy search scheduled every ten minutes that overloaded the system!
Anyway, we used higher timeout value.
Bye.
Giuseppe

0 Karma

kedjjang
Explorer

cusello hello,

We are also having the same problem as you.
What configuration files and options have you modified?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi vrmandadi,
At first, using Splunk Monitoring Console, see if there are peaks of CPU.
Then see if there are scheduled searches and/or accelerated searches [Settings -- Searches, Reports and Alerts] and if someone of them are scheduled at the same time of the peaks.
Then see if you can optimize these searches: see if there are joins or transactions, or accelerations, in other words: there isn't a configuration file to modify, you have to find the critical searches and then optimize them.

I can report you my experience:
in my system I found that there was a peak every then minutes,
watching scheduled search I found that there was a very heavy accelated search that started every then minutes!
Than I planned in a different way this search (I transformed my search in a scheduled report running once a day in the night) and my system restarted to work well!

I hope to be useful for you.

Bye.
Giuseppe

0 Karma

jensonthottian
Contributor

The search head you are on is not a able to connect with peer (https://foobar237.xxx.com:8089).

Make sure you set distributed search properly: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Configuredistributedsearch

If you getting too much of these then you can edit distsearch.conf. Also check the splunkd.log on foobar237.xxx.com to see what is going wrong there.

Check out these settings in distsearch.conf:

connectionTimeout = 
  * Amount of time in seconds to use as a timeout during search peer connection establishment.

sendTimeout = 
  * Amount of time in seconds to use as a timeout while trying to write/send data to a search peer.

receiveTimeout = 
  * Amount of time in seconds to use as a timeout while trying to read/receive data from a search peer.

http://docs.splunk.com/Documentation/Splunk/latest/admin/distsearchconf

vrmandadi
Builder

is there anywhere to resolve it

0 Karma

nawazns5038
Builder

Should we check it on the search head or the peer ?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

On the search Head. (Settings-> Distributed Search-> Search peers)

somesoni2
SplunkTrust
SplunkTrust

The search head lost connectivity to the host/search peer/indexer mentioned by the URL.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...