I was troubleshooting why peers show as "Pending" often in the cluster master web UI. In troubleshooting I ran 'ss |less' and via TCP, I found the master connecting on odd ports and vice versa. Here's a "sanitized" example:
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.06:41346
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.04:47714
tcp ESTAB 0 0 172.indexercluster.master.ip:40738 172.indexercluster.member.015:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:53218 172.indexercluster.member.010:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:57761 172.indexercluster.member.018:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:60002 172.indexercluster.member.012:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:54722 172.indexercluster.member.021:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:57434 172.indexercluster.member.014:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.010:40392
tcp ESTAB 0 0 172.indexercluster.master.ip:57484 172.indexercluster.member.014:8089
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.018:39212
tcp ESTAB 0 0 172.indexercluster.master.ip:44492 172.indexercluster.member.013:8089
Is this normal communication or something strange?
Not sure I've noticed this before, so I wanted to see if anyone else has seen this.
Thanks
With a TCP connection the 'client' connects to the 'server' on a known or target port.
In the case of Splunk this is 8089. However since TCP is a bidirectional protocol it needs to tell the server which port to reply back on.
In most c/s architectures, the port the client chooses is a random 'high-number' port - 41346 from the top line is one such example.
The top line, is a connection from 'member6' using 41346 as its src port to the dest port 8089 on the master.
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.06:41346
The third line, is a connection from the master with src port 40738 to the dest port 8089 on 'member15'
tcp ESTAB 0 0 172.indexercluster.master.ip:40738 172.indexercluster.member.015:8089
What your seeing is totally normal TCP communication patterns.
With a TCP connection the 'client' connects to the 'server' on a known or target port.
In the case of Splunk this is 8089. However since TCP is a bidirectional protocol it needs to tell the server which port to reply back on.
In most c/s architectures, the port the client chooses is a random 'high-number' port - 41346 from the top line is one such example.
The top line, is a connection from 'member6' using 41346 as its src port to the dest port 8089 on the master.
tcp ESTAB 0 0 172.indexercluster.master.ip:8089 172.indexercluster.member.06:41346
The third line, is a connection from the master with src port 40738 to the dest port 8089 on 'member15'
tcp ESTAB 0 0 172.indexercluster.master.ip:40738 172.indexercluster.member.015:8089
What your seeing is totally normal TCP communication patterns.
Thanks, was just making sure it wasn't something abnormal