We've installed and are evaluating Splunk Enterprise 6.0 in a Windows environment (desktops are running Windows 7 Professional x64, and servers are running Windows Server 2008 R2). We're very happy with Splunk so far, but have run into a puzzling problem. We have two networks with non-overlapping subnets, network X on 10/8 and network Y on 172.30/16. The Splunk indexer is running on a server that has two NICs, one connected to the X network with static address 10.0.0.50, and one connected to the Y network with static address 172.30.0.50. So, as expected, the indexer receives data from forwarders on both networks. However, each forwarder is attempting to communicate with the indexer on both networks, even though each forwarder is only connected to one network at a time. For example, a forwarder on a desktop on network X will try to communicate with the indexer using the indexer's network Y address 172.30.0.50; needless to say, that communication fails, but the desktop will also try to communicate with the indexer using the indexer's X address 10.0.0.50, which succeeds. All of the forwarders eventually get their data to the indexer, but there are tens of thousands of packets that end up being dropped every day, and the forwarders' metrics.log files are full of lines like:
log/splunk/metrics.log.1:03-10-2014 15:04:37.230 -0400 INFO StatusMgr - destHost=[indexer server name], destIp=[IP on wrong network], destPort=9997, eventType=connect_fail, publisher=tpcout, sourcePort=8089, statusee=TcpOutputProcessor
Expecting that the problem was with our local DNS servers (running on our domain controllers, one of which is the same server that the indexer is on), we checked and confirmed that we had round robin disabled and netmask ordering enabled. So when we nslookup the indexer server on any machine on network X, we receive the answers 10.0.0.50 and 172.30.0.50 (with the correct network first), and when we nslookup the indexer server on any machine on network Y, we receive the answers 172.30.0.50 and 10.0.0.50 (in the opposite order, still with the correct network first). So it appears that our DNS servers are working correctly.
We do not have a deployment server. When we installed the forwarders initially, we specified the plain name of the indexer server, but uninstalling a forwarder, reinstalling it and specifying the fully qualified name of the indexer (indexer.domain.local) doesn't help. For many desktops we can't specify the indexer's IP address when installing the forwarder, because those desktops can switch between the two networks using a Black Box 2-to-1 CAT6 manual switch.
This may not sound like a serious problem, just a waste of network bandwidth, since the forwarders eventually get their data to the indexer, but it's an issue for us, because our firewall is dropping and logging the misdirected packets, and they're adding a lot of noise to the security monitoring we're doing on the firewall. Does anyone have any ideas?
... View more