We have configured an intermediary hf, C and 2 HFs - A and B connecting to C.
The HF A is able to establish connection and send data to HF C over ssl but the HF B is not.
We need your help fixing this.
Forwarder A can SSL-communicate with intermediary forwarder C.
Forwarder B can NOT SSL-communicate with intermediary forwarder C.
A and C are in the same chassis/compartment and B is not.
Log messages
A has no problem:
02-19-2020 23:38:26.344 +0000 INFO TcpOutputProc - Connected to idx=1.2.3.4:9997, pset=0, reuse=0. using ACK.
B gets an error below;
02-19-2020 23:41:09.861 +0000 ERROR TcpOutputFd - Connection to host=1.2.3.4:9997 failed. sock_error = 104. SSL Error = error:00000000:lib(0):func(0):reason(0)
It turned out to be a firewall issue and to get to the point, here's what I did ;
i) Configuration : Check if there's any configuration missing -> All good according to the docs.
ii) Certs : Use the same certs and configuration from A for B so that it can eliminate the mistakes during the cert creation or configs on forwarder B. -> still gets error.
iii) Certs: Use default certs and see if it is still the same -> still the same issue happens. This tells us it could be an issue caused by outside splunk entities.
iv) splunk cmd openssl s_client -connect C:9997 -cert from A and B to see what happens.
HF A - OK
HF B still fails.
v) Network entities sitting between B and C could break sessions: Capture tcp packets to get some hints for sock_error=104, to understand why the other end sends tcp reset (104)
Capture it from both end as network entity in between is suspicious.
v-1)On B, tcpdump -i any -ta -s0 host C and port 9997 -w output.pcap
v-2)On C, tcpdump -i any -ta -s0 host B and port 9997 -w output.pcap
v-2) pcap from HF C says differently - HF B sends a RST to HF C
Judging by v-1) & v-2) outcomes there must be a network entity that breaks the session by sending RSTs to both ends.
This has been fixed with the help of network team by fixing a firewall configuration.
It turned out to be a firewall issue and to get to the point, here's what I did ;
i) Configuration : Check if there's any configuration missing -> All good according to the docs.
ii) Certs : Use the same certs and configuration from A for B so that it can eliminate the mistakes during the cert creation or configs on forwarder B. -> still gets error.
iii) Certs: Use default certs and see if it is still the same -> still the same issue happens. This tells us it could be an issue caused by outside splunk entities.
iv) splunk cmd openssl s_client -connect C:9997 -cert from A and B to see what happens.
HF A - OK
HF B still fails.
v) Network entities sitting between B and C could break sessions: Capture tcp packets to get some hints for sock_error=104, to understand why the other end sends tcp reset (104)
Capture it from both end as network entity in between is suspicious.
v-1)On B, tcpdump -i any -ta -s0 host C and port 9997 -w output.pcap
v-2)On C, tcpdump -i any -ta -s0 host B and port 9997 -w output.pcap
v-2) pcap from HF C says differently - HF B sends a RST to HF C
Judging by v-1) & v-2) outcomes there must be a network entity that breaks the session by sending RSTs to both ends.
This has been fixed with the help of network team by fixing a firewall configuration.