What is the minimum network bandwidth required for Splunk forwarding?

[maverick]

I plan to setup Splunk Forwarders to push Windows Events and also some linux events to my central Splunk indexer. Need to know what the minimum network bandwidth pipe is required for this?

[dwaddle · Accepted solution]

This is a serious "it depends" question. A forwarder's bandwidth requirements depends greatly on the event volume going through it.

A Splunk Light Forwarder has a built-in governor on its bandwidth usage. The default for a single light forwarder is 256 kilobytes per second (set in $SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf)

Based on that alone, one could argue that each forwarder needs as much as 2 Megabits/sec of bandwidth - assuming the event volume is high enough to consistently use that much bandwidth. But if your event volume is sufficiently low, you don't need nearly that much...

The questions you should be asking yourself, based on the data you have is how many forwarders, producing how many messages per second each, with an average message size of how many bytes? Then you can begin to estimate your actual bandwidth needs.

[mattymo]

I upvoted @dwaddle & @bwooden as it truly depends on many factors, like number of events, number of forwarders, as well as stakeholder expectations/requirements, and how much time and money you have!

I thought it worth adding that our reference specs, while not perfect, do cite 1GB NICs.

https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware

So as the others have stated the forwarders have configs that can control the flow of data, it really is key to have the core of your Splunk deploy (indexers, search heads, intermediate forwarders) to have performant pipes, and I’d say that 1GB NICs would be the minimum.

[cweinhaupl]

Is there any compression used by the Forwarder on text based log files?

[mdessus_splunk]

Yes, there is a crompressed parameter in the outputs.conf file for non-ssl connections, or useClientSSLCompression for the SSL ones.

[tpaulsen]

Yes, i would like to know this as well:

If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data? – Marinus Apr 30 at 8:31

[gkanapathy]

could you post this as a separate question?

[bwooden · Accepted solution]

There may not be a one size fits all answer. If you deploy a light weight forwarder it limits itself to 256 kilobytes / second by default. This is tunable in the limits.conf file

[thruput]
maxKBps = 256

If the forwarder is configured too low it may fall behind a bit during times of elevated event generations.

[dwaddle · Accepted solution]

This is a serious "it depends" question. A forwarder's bandwidth requirements depends greatly on the event volume going through it.

A Splunk Light Forwarder has a built-in governor on its bandwidth usage. The default for a single light forwarder is 256 kilobytes per second (set in $SPLUNK_HOME/etc/apps/SplunkLightForwarder/default/limits.conf)

Based on that alone, one could argue that each forwarder needs as much as 2 Megabits/sec of bandwidth - assuming the event volume is high enough to consistently use that much bandwidth. But if your event volume is sufficiently low, you don't need nearly that much...

The questions you should be asking yourself, based on the data you have is how many forwarders, producing how many messages per second each, with an average message size of how many bytes? Then you can begin to estimate your actual bandwidth needs.

[yannK]

when the forwarder pauses :

It will be able to resume where it stopped for all the persistent inputs
- log file monitoring (As long as the files do not rotate out of reach from splunk.)
- wineventlogs (using a counter)
- etc ...
- another forwarder (it will pause too in chain)

but the forwarder will not be able to resume from non persistent inputs, once the queue buffer is full, the new events will be dropped.
- tcp/udp inputs (like syslog)
- scripted inputs
the workaround are to use persistent queues or have your logs written to file, then monitor them.

[Jarohnimo]

You risk data "lost in flight" if 1) your network isn't suitable (10gb is preferred)
2) your maxkbps is too low
3) parsing queue is too low

[Marinus]

If you set it low and the queue fills up on the forwarder, will it back off until it's sent the queue? Could you loose data?