Getting Data In

Is it possible to disable Load Balancing between Universal Forwarders and the Heavy Forwarder?

kylerose
Explorer

We have many systems with Universal Forwarders sending to a dedicated Heavy Forwarder. We would like to put a 3rd party load balancer in-between the UFs and HF to assist with scaling and automation. Is it possible to disable load balancing between the UFs and the HF ? Can I set autoLB = false to do this?

1 Solution

dwaddle
SplunkTrust
SplunkTrust

I don't think this is a supported configuration. To my knowledge, Splunk support does not support the use of a hardware load balancer between forwarders and indexers. What you would gain from one is also perhaps dubious.

When you configure Splunk to use a single IP (or a DNS name that resolves to a single IP), then the forwarders connect their TCP session and never disconnect. This would mean that each forwarder would, for the life of its process, always send data to a single indexer. (Unless, of course, the TCP connection was interrupted for some reason).

When Splunk has multiple IPs, or a DNS name that resolves to multiple IPs, then its round-robin will connect, send data, and disconnect. This way, you get a more even spread of data across indexers.

I just don't think a physical load balancer gives you anything good here. Actually, it probably makes things worse.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

I don't think this is a supported configuration. To my knowledge, Splunk support does not support the use of a hardware load balancer between forwarders and indexers. What you would gain from one is also perhaps dubious.

When you configure Splunk to use a single IP (or a DNS name that resolves to a single IP), then the forwarders connect their TCP session and never disconnect. This would mean that each forwarder would, for the life of its process, always send data to a single indexer. (Unless, of course, the TCP connection was interrupted for some reason).

When Splunk has multiple IPs, or a DNS name that resolves to multiple IPs, then its round-robin will connect, send data, and disconnect. This way, you get a more even spread of data across indexers.

I just don't think a physical load balancer gives you anything good here. Actually, it probably makes things worse.

kylerose
Explorer

I understand what you mean, but in a Cloud environment it makes it easier to automate configurations and scale using a "hardware" LB as opposed to Splunk built in load balancing. Maybe Splunk needs to catch up to and add more options for running in dynamic/ephemeral environments.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

I'll ask, are you honestly adding/removing ephemeral heavy forwarders at a dynamic frequency? I understand that most "web" people consider round-robin DNS to be an inferior solution because of webapp state, but Splunk does WELL with it. Most DNS servers can do Dynamic DNS as well, which is fully scriptable to where you can have your dynamic / ephemeral systems added (or removed) from the DNS record automatically as things change. Amazon's Route53, for instance, has a nice API for changing RRSets.

You want to make certain that whatever environment you build is supported to the degree that if something goes wrong, you are able to get help. Putting an F5 or an AWS ELB etc in between forwarders and indexers (or forwarders and intermediate forwarders) for the forwarded data is not supported by Splunk support.

If you think Splunk needs additional features in this area, then I would suggest filing a feature request.

0 Karma

kylerose
Explorer

The UFs can be extremely ephemeral depending on the service. The HFs are restacked around twice a month to apply security patches. We actually are looking into the Route53 API, but this will require some work as opposed to using an ELB (which is why I asked the question). Thanks for your answer!

0 Karma

merp96
Path Finder

Yes by setting "autoLB = false" in all Universal Forwarders outputs.conf, you could disable Splunk's load balancing and make use of third party load balancer.

At the moment since all Universal Forwarders are sending to a single heavyforwarder "autoLB = false" does not make any difference. But when you start scaling up the number of Heavyforwarders then yes you could use "autoLB = false" in Universal Forwarders outputs.conf to make use of your own balancers.
Also ensure that you use TCP to ensure that packets are not broken. As you are planning to use a third party load balancer better to use TCP Protocol to send from UF ->HF

merp96
Path Finder

However I would strongly suggest to go with Splunk's load balancing as it works fine and is already automated and allows scaling, unless you have a custom requirement

Please refer the answer in this URL :- http://answers.splunk.com/answers/222522/loadbalancer-or-intermediate-forwarder.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...