Splunk UF forwarding to a unidirectional data diode which then forwards logs to Splunk server. No longer receiving logs from UF. (air gapped environment)

[Rshoufi]

Here's a quick rundown of the environment: Virtual Machines (linux splunk instances), No internet connection, air gapped environment that uses a unidirectional data diode. In this environment there is going to be very little data which is why there is just a single instance of Splunk (IDX, SH, and LM) and 1 universal forwarder. Oh, and for those of you reading along and new to splunk/networking and are asking "wtf is a data diode?" here is a short explanation "The concept of a data diode is simple: specifically designed hardware circuitry within which it is only possible for data to flow in one direction" In this case the data flows from the UF via UDP 514 to to side A interface of the diode with an example IP of (192.168.10.15). This interface is supposed to then push all of that forwarded data out of side B of the data diode which then pushes that data to the splunk server which is configured to listen on local input TCP 514 because I was told by the engineering team that's just how it was and didn't receive an explanation as to why one side was configured UDP and the other TCP.

The problem I have is ever since we added the diode aspect to the environment, Splunk no longer receives logs and I have no idea where to begin troubleshooting. The IP addresses in the UF and Splunk server have been corrected to reflect the change of location in the environment and rebinded the new IP address etc etc. Now, I don't know if this is because of a misconfiguration on my end of things or because the diode itself isn't properly setup yet. But from what I've explained am I understanding this is how the configuration is supposed to be in Splunk?

** Configure the universal forwarder to forward the syslog-ng data to the interface/IP of the data diode via UDP 514
** Then have the diode push that information outbound towards the splunk server
** Splunk is now listening on TCP 514 for the incoming syslog-ng data.

                       Side A of Diode    (air gap)   Side B of Diode

UF(x.x.10.25) -----> x.x.10.15 --> ||||| x.x.13.15 ----------> Splunk server(x.x.13.26)

Splunk should see this data as being sent from the data diode and not the universal forwarder correct? I would expect the logs to also include the IP addresses of both sides of the data diode as well as the IP of the UF..

Am I understanding this correctly? Or am I way off base?

[jpappe]

This is how I got the SUF to send Windows logs to Splunk Enterprise through my data diode. On the SUF side, I modified my \etc\system\local\output.conf by adding this line:

sendCookedData = false

I then changed the server and tcpout-server to my protected diode interface ip and port:

server = 192.168.1.11:8997
[tcpout-server://192.168.1.11:8997]

(Note: You could skip those last two settings if you install the SUF and specify the IP and port in the customized settings.)

On the unprotected diode interface, I send the data to the Splunk Enterprise on non-SUF port (anything other than 9997).
I added a TCP data input for that port and set the source to tcp-raw.

[jpappe]

My experience with transmitting syslog on udp 514 thru my data-diode was rather straightforward. Using the UF has been more of a challenge. Using Wireshark I can see the DD and Splunk negotiating the connection, but then the Splunk side seems to want to negotiate an S2S connection with the UF client which the DD obviously rejects. Not sure how a heavy forwarder works but will investigate. Rshoufi, what was your final resolution?

[jpappe]

Looking at the SUF logs and packet captures from both ends shows me that the SUF side stalls as it doesn't see the S2S control message from Enterprise. When this happens, it pauses for 100 seconds then posts a warning in the splunkd log:

WARN TcpOutputProc - Cooked connection to IP = 192.168.1.11:XXXX timed out.

FWIW I tried another similar forwarding client to send my windows logs via tcp thru the DD to Splunk Enterprise, and it works fine. Not sure what magic happens in the S2S handshake but wish I could turn it off.

[woodcock]

Make sure all your syslog stuff is UDP (not TCP, which is a 2-way protocol). The S2S protocol that splunk uses is also TCP-ish so you have to send directly to the Indexers with a UDP listener there (a NOT best practice) for this to work (so that you are not using S2S)

[Rshoufi]

I think you just confirmed what I suspected earlier. Unfortunately, I cannot send directly to an indexer which means I would need to have a heavy forwarder setup instead of the UF they previously installed.
Each side of the diode is configured to run a different protocol at the time. As of now the information forwarding from the UF to the diode is via TCP (3rd party technically) the other side of the node pointing to the Splunk indexer is configured to listen on UDP.

Thanks for the advice, going to get it sorted now and will report back.