Getting Data In

Unable to distribute to peer oddness

tgiles
Path Finder

Hi,

I have two pooled search heads which search a couple of indexers. heads connect across a public IP address to the indexers. The indexers have a private IP address name in their configuration. For example:

  • Indexer 1: peer name 192.0.32.10:8089, splunk server name 10.999.20.5
  • Indexer 2: peer name 192.0.32.11:8089, splunk server name 10.999.20.6

intermittently, a search head will throw an error bar at the top, reporting a connection problem:

Unable to distribute to peer named
192.0.32.10:8089 at uri https://192.0.32.10:8089 because peer
has status = "Down".

The message is confusing because it's giving the 'peer name' as the wrong thing. If it was an actual error, i'd assume it would call the problem peer by the correct peer name (in this instance, 10.999.20.5) and not the external IP.

I wrote up a little python scripts to constantly make socket network connections to the affected indexer. Even when Splunk reported it couldn't connect to the indexer, my script had no issues opening network connections.

Double-checked all the splunk indexer configuration files, just in case I have a bad configuration somewhere- everything looks clean. networking guys reported no issues with the firewall logs. Indexer itself looks fine, logs are coming in, no problems observed in the logs on it.

have any ideas on how to troubleshoot an indexer connection problem when I can't replicate it using another method? any particulars to look for in the logs?

Thanks for your input

Tags (3)

lmyrefelt
Builder

Well its hard to say without having a look at your confs. But i "useally" get this if the indexer is under heavy load and can responed to all requests. When in searchhead pooling the performance of the central located share (nfs/cifs/ whatever) is important as well since the results and whatnot is saved if not only temporary there.

For multihome splunk (indexer) instanses there seems to be some things you can look at.

$SPLUNK_HOME/etc/splunk-launch.conf
SPLUNK_BINDIP=

$SPLUNK_HOME/etc/system/local/web.conf
mgmtHostPort =
server.socket_host =

0 Karma

lmyrefelt
Builder
0 Karma

lukereeves
Engager

This is happening to me as well, I guess you never found a resolution? I'll make a support ticket for this.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...