We know that following recommendations that the rule of thumb for indexers is one indexer per 100GB indexed per 24 hours when not performing replication.
Is there a rule of thumb for Heavy Forwarders when performing filtering/routing/time zone functions, however not indexing locally?
i.e. UF (monitor) --> HF (filtering/routing/time zones) --> IDX (indexing)
Could I estimate that I would need a new Heavy Forwarder per 200GB per 24 hours?
http://docs.splunk.com/Documentation/Splunk/6.1.3/Deploy/Summaryofperformancerecommendations
This is an awesome question, but I think it's more a function of configuration and environment than anything else. Best thing I can suggest is watch your performance counters (CPU, RAM, I/O, Splunk Queues, Network) for your box and gauge growth from there.
We currently have HF doing some preprocessing for sourcetyping on syslog, but it is largely a bulk forwarder for UF & syslog. We are logging significantly more than 200GB per day per HF and don't seem to have issues (our issues are generally self created or particular to our environment).
The rule of thumb is actually now 1 indexer per 250 GB/day. Not per 100 GB/day. You can probably estimate a HF as being able to max between 8 and 20 MB/sec, depending on the specifics of the server, the data, and any processing you might be doing.
Yeah, it was, though it was half-updated.
This is an awesome question, but I think it's more a function of configuration and environment than anything else. Best thing I can suggest is watch your performance counters (CPU, RAM, I/O, Splunk Queues, Network) for your box and gauge growth from there.
We currently have HF doing some preprocessing for sourcetyping on syslog, but it is largely a bulk forwarder for UF & syslog. We are logging significantly more than 200GB per day per HF and don't seem to have issues (our issues are generally self created or particular to our environment).
Thanks for sharing this helpful info. It's always good if we can avoid going back to management and saying "we underestimated the volume it could handle."