Getting Data In

"search head Timed out waiting for peer" How can I check the network health between a search head and indexer?

nivedita_viswan
Path Finder

Our environment consists of 1 indexer and 1 search head. Our indexer is currently indexing close to 400GB per day, since we are catching up on historical data. In another week, this should reduce to about 20GB per day.
Meanwhile, we are running a few saved searches on the search head, which would normally run for a few hours. However, we always see the error:

Timed out waiting for peer xx-xxxx-xxx. If this occurs frequently, receiveTimeout in distsearch.conf may need to be increased. Search results might be incomplete!

I have increased receiveTimeout to 900s. I am planning on adding the following stanza to distsearch.conf to reduce the knowledge bundle size:

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

I know that there may be network issues that are causing the problem. Are there any commands I can use to check the network health between the search head and indexer?
Any other suggestions to avoid this message would be welcome.

1 Solution

nivedita_viswan
Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

View solution in original post

nivedita_viswan
Path Finder

It turns out the problem was the nature of the query itself. Since the query was searching for sparse events, there were cases where it would run for 900s, and actually have nothing to return.
Increasing the timeout to a much larger value solved the problem

brodriguez
Splunk Employee
Splunk Employee

Could you please specify which timeout setting did you increase?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

You're digging up an 8 years old thread. I wouldn't expect an answer from its original participants...

0 Karma

dflodstrom
Builder

Have you covered the basics with ping, traceroute, and telnet?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...