Is it normal for an indexer cluster master to connect to peers on odd ports?

[tkw03]

I was troubleshooting why peers show as "Pending" often in the cluster master web UI. In troubleshooting I ran 'ss |less' and via TCP, I found the master connecting on odd ports and vice versa. Here's a "sanitized" example:

Netid  State      Recv-Q Send-Q Local Address:Port                 Peer Address:Port 
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.06:41346
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.04:47714
tcp    ESTAB      0      0      172.indexercluster.master.ip:40738                172.indexercluster.member.015:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:53218                172.indexercluster.member.010:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:57761                172.indexercluster.member.018:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:60002                172.indexercluster.member.012:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:54722                172.indexercluster.member.021:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:57434                172.indexercluster.member.014:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.010:40392
tcp    ESTAB      0      0      172.indexercluster.master.ip:57484                172.indexercluster.member.014:8089
tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.018:39212
tcp    ESTAB      0      0      172.indexercluster.master.ip:44492                172.indexercluster.member.013:8089

Is this normal communication or something strange?

Not sure I've noticed this before, so I wanted to see if anyone else has seen this.

Thanks

[nickhills · Accepted solution]

With a TCP connection the 'client' connects to the 'server' on a known or target port.
In the case of Splunk this is 8089. However since TCP is a bidirectional protocol it needs to tell the server which port to reply back on.
In most c/s architectures, the port the client chooses is a random 'high-number' port - 41346 from the top line is one such example.

The top line, is a connection from 'member6' using 41346 as its src port to the dest port 8089 on the master.

tcp    ESTAB      0      0      172.indexercluster.master.ip:8089                 172.indexercluster.member.06:41346

The third line, is a connection from the master with src port 40738 to the dest port 8089 on 'member15'

tcp    ESTAB      0      0      172.indexercluster.master.ip:40738                172.indexercluster.member.015:8089

What your seeing is totally normal TCP communication patterns.

[tkw03]

Thanks, was just making sure it wasn't something abnormal