intermediary heavy forwarder tier

[a212830]

Hi,

As we begin to roll-out hundreds and thousands of universal forwarders, I was wondering if it made sense to create a layer of heavy forwarders, to take some load off the indexers. Has anyone done this? Does it make sense?

[a212830]

I will have multiple indexers, but I also want to do whatever is possible to reduce the load on the indexers.

[sciurus]

I recently had this discussion, but it was more around using regional heavy forwarders that feed in to a central indexer. We had existing heavy forwarders for syslog collection, so had to decide whether or not to send universal forwarder data towards them, or direct to the indexer.

Pros of using HF / cons of not using HF:

1. Traffic can be uncompressed / unencrypted towards the HF, saving CPU, the HF can then compress and/or encrypt it for bandwidth and security, as it's got dedicated CPU
2. The HF can drop and route events
3. The HF can re-write data, such as credit cards, before sending it to the IDX
4. The HF can cook the data before sending it to the IDX

Pros of not using HF / cons of using HF:

1. Each UF has the same output configuration, making it easier to manage (deployment server mitigates this)
2. If a HF goes down (planned or unplanned), logs from the UFs still arrive
3. The general philosophy of Splunk seems to be it's better to go direct
4. If you have 1000 UFs doing auto load-balancing between two IDXs, the split will be very close to 50/50 as they all switch around. If you have 4 or 5 HFs, it's entirely possible you could end up with all of them pointing to one IDX and hammering it before they switch out.

Can it be done? Yes. Should it? That's entirely up to you, and hopefully this helps you ask the right questions for your scenario.

[somesoni2]

What about having multiple indexers to share the load? http://docs.splunk.com/Documentation/Splunk/6.0/Forwarding/Setuploadbalancingd