Splunk forwarder behind firewall. Splunk server behind another firewall.

[bbegyspotlite]

So I configured a universal forwarder on one of our production web servers, telling it to go forward windows logs to MY-OFFICE-IP:4990. Of course, the office has a firewall (SonicWall). I configured the SonicWall to allow traffic from MY-PRODUCTION-IP to MY-SPLUNK-SERVER-IP on ports 4990-4999.

I can see the connections opened on my firewall, but no logs appear in splunk. (thanks to splunk monitoring our SonicWall's syslogs, I have stuff like this:

Mar 6 09:10:11 10.1.10.1 id=firewall sn=0017C5ABE018 time="2013-03-06 09:10:11" fw=MY-OFFICE-IP pri=6 c=262144 m=98 msg="Connection Opened" n=1402293 src=MY-PRODUCTION-IP:36714:X1 dst=MY-OFFICE-IP:4990:X1 proto=tcp/4990

Mar 6 09:10:10 10.1.10.1 id=firewall sn=0017C5ABE018 time="2013-03-06 09:10:10" fw=MY-OFFICE-IP pri=5 c=64 m=36 msg="TCP connection dropped" n=39953 src=MY-PRODUCTION-IP:36714:X1 dst=MY-OFFICE-IP:4990:X1 proto=tcp/4990

Do I need to open any ports on my Production firewall to allow the splunk handshake to work? Is there a FAQ or other documentation on this use case that I should have consulted before asking?

[jsundberg_splun]

1st thing I'd check, since this is a Windows box, and you've already thought of the other two FW's in the path of the data, is to check that the actual Windows firewall itself isn't on. It's easily overlooked.

Then on the Windows server you've installed the Forwarder on, you should be able to just open

 c:\program files\splunkuniveralforwarder\etc\system\local\outputs.conf 

which should read similar to this:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = splunk.index.ip:port (default is 9997, looks like you used 4990 based on you're SW FW log above)

[tcpout-server://splunk.index.server.ip:port]

Then, on the main receiving Splunk Indexer:
Go into Manager > Data/Forwarding and Receiving > Configure Receiving > Add New

Then tell Splunk what port to listen on, in this case I assume 4490.
If you review these items you should start seeing your indexed count go up.

For reference here's the UF docs:

http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Deploymentoverview

http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Introducingtheuniversalforwarder