Splunk Dev

Usage intermediate forwarders with load balancer

marcheyde
New Member

End of last year we migrated from Splunk 6.5.3 to 7.1.3

The universal forwarders on the different source systems delivering our inputs,
send data via a load balancer to 2 intermediate forwarders, connected with our 6 indexers.
That setup was recommended to us a few year ago (by a splunk partner) with the initial setup of our system.

We found information indicating that the best setup recommended today is a direct connection between universal forwarders of the source systems and the indexers of our splunk cluster (no intermediate forwarders with a load balancer).

Anyone who can comment on this?

0 Karma

koshyk
Super Champion

Good question and I could see lot of variations/personal views on this setup. But please find my experience and some inputs

  1. Option for Best performance and resilience => UF to Indexers. So if you have 120UF's out there, that means the system directly load balances the data into 6 indexers. that means each indexer get's 120::6 ratio of buckets each at a time assuming if it indexes just one file and splits the load evenly. When you search, the buckets are split and better for retrieving results
  2. Option for more security. Though Lot of people go with above option, my view is it is NOT good to expose your indexer directly to client systems. Opinions may vary, but I would like to separate it with another layer like a Heavy forwarder layer or another UF layer.
  3. Your setup of putting load balancer in front of Intermediate forwarders is NOT a good practice. It makes the whole splunk load balancing get wasted. The above is good ONLY if the load balancer is enabled for syslog data (and not from UF)
  4. Compromise solution (option I'm leaning for) => Have an intermediate layer (UF or HF) and increase the number as much as possible with money allowance. Secure it and allow only specific ports and data collection only. Don't expose management ports. Assume you got 12 Intermediate forwarders. and if you have enough CPU/memory, enable multiple pipelines (say 2) which makes it similar to 24 Intermediate fwd. So assuming 120 clients sends to 24 Intermediate fwds which then sends to 6 indexers. So you still get a even split of 24::6 ratio. This way you can secure and load balance the data. This option also gives advantage of opening Firewall to Cloud services/Internet dataset without ever exposing Indexers
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...