Load Balancer between Heavy Forwarder and Indexers

[ansif]

We have implemented 3rd party hardware load balancer between heavy forwarder (which is different network - Customer site) and Indexer (at our site) but it caused event delay.

Thus we removed the load balancer (taking into account that the splunk not recommends) and it is working fine.However we really require to implement load balancing directly to indexer to avoid single point of failiure(Heavy forwarder at our site).Any recommendations?

[nickhills · Accepted solution]

I'm not sure what the driver behind the use case is (cost vs availability), but with a 2meg link and a load balancer I am assuming its not speed.

You could put some (or all) of your indexing & receiving elsewhere like Amazon. You can then build out a receiving platform on AWS to suit your needs, without the constrains you have on NAT/bandwidth.

Your options then:
-forward events from AWS to your On-Prem indexers
-run a local search head, and remote to AWS hosted indexers
-run it all in AWS, and use https to access AWS hosted search heads & indexers.
(I overlook networking complexity, vpcs, vpns etc - but it really depends on your exact needs & constraints)

[harsmarvania57]

Hi @ansif,

I didn't get your design properly. So based on my assumption you can think about below design.

UF (Remote)  -> HF1 (Remote) & HF2 (Remote) -> Dedicated Network Link (Maybe over internet) -> HF1 (Local) & HF2 (local) -> Multiple Indexers                                                                                                                

Splunk by default ships load balancer to send data to multiple splunk instances, please refer http://docs.splunk.com/Documentation/Forwarder/7.0.0/Forwarder/InstallaWindowsuniversalforwarderfrom... to setup load balancing on UF ans HF and you need to use useACK parameter so that data will not be lost (refer doc http://docs.splunk.com/Documentation/Splunk/7.0.1/Forwarding/Protectagainstlossofin-flightdata ) but this will create more network overhead and sometime you will receive duplicate data as well if acknowledgement response didn't reach back to UF due to packet drop in network but those data already indexed but UF assumes that it didn't indexed so it will send same data to another HF which ends up with duplicate the data.

So in above design UF Sends data to HF1(Remote) and HF2(Remote) with auto load balancing and acknowledgment ( useACK), now assume that HF1(remote) went down so when UF will send data to HF1(Remote) it will not provide acknowledgement response back to UF so UF will send those data to HF2(Remote). Same way you can setup acknowledgement and autoLB between HF(Remote) & HF(Local) and HF(Local) & Indexers.

I hope this helps.

Thanks,
Harshil

[nickhills]

The obvious choice is to run more than one receiver (* by which I mean a receiving HF)on your site.
The customer HF then is configured to send to n+ targets.

Of course, this does not protect against network disruption, although unless your existing LB works with multi-SP perhaps this is not too much of an issue?

If you do have multiple service providers you could locate a receiver on both, which should give you pretty good coverage.

Another alternative (depending on where the source data actually originates from) might be to use the Http Event Collector instead of Heavy Forwarders - this absolutely does support Load Balancing (as its just http), and is a jolly efficient means of collecting data from desperate network locations, and has excellent support for scaling and HA!
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/UsetheHTTPEventCollector

[ansif]

The HEC is new to me.Thanks for that.

More than one receiver means, need to have multiple NAT IP's that is again an overhead.

Unfortunately the connection is on a dedicated MPLS of 2Mbps.

Adding ,we are getting several types of data,so it is not possible with HEC and several filtering and annonymizes is doing at customer's HF.