Getting Data In

Estimating number of Heavy Forwarders based on event volume?

bandit
Motivator

We know that following recommendations that the rule of thumb for indexers is one indexer per 100GB indexed per 24 hours when not performing replication.

Is there a rule of thumb for Heavy Forwarders when performing filtering/routing/time zone functions, however not indexing locally?

i.e. UF (monitor) --> HF (filtering/routing/time zones) --> IDX (indexing)

Could I estimate that I would need a new Heavy Forwarder per 200GB per 24 hours?

http://docs.splunk.com/Documentation/Splunk/6.1.3/Deploy/Summaryofperformancerecommendations

1 Solution

ltrand
Contributor

This is an awesome question, but I think it's more a function of configuration and environment than anything else. Best thing I can suggest is watch your performance counters (CPU, RAM, I/O, Splunk Queues, Network) for your box and gauge growth from there.

We currently have HF doing some preprocessing for sourcetyping on syslog, but it is largely a bulk forwarder for UF & syslog. We are logging significantly more than 200GB per day per HF and don't seem to have issues (our issues are generally self created or particular to our environment).

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

The rule of thumb is actually now 1 indexer per 250 GB/day. Not per 100 GB/day. You can probably estimate a HF as being able to max between 8 and 20 MB/sec, depending on the specifics of the server, the data, and any processing you might be doing.

bandit
Motivator
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Yeah, it was, though it was half-updated.

ltrand
Contributor

This is an awesome question, but I think it's more a function of configuration and environment than anything else. Best thing I can suggest is watch your performance counters (CPU, RAM, I/O, Splunk Queues, Network) for your box and gauge growth from there.

We currently have HF doing some preprocessing for sourcetyping on syslog, but it is largely a bulk forwarder for UF & syslog. We are logging significantly more than 200GB per day per HF and don't seem to have issues (our issues are generally self created or particular to our environment).

bandit
Motivator

Thanks for sharing this helpful info. It's always good if we can avoid going back to management and saying "we underestimated the volume it could handle."

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...