Reliability of a forwarder

[kris2000]

Hello everyone ,

We are planning to have a Splunk setup as below:

LightForwarders -> Forwarders -> Indexers

Assuming that all the above instances are on different physical hosts -

When a forwarder goes down(either gracious or kill) is there a possibility that the data/events could be lost? This is assuming when all the values for parameters like blockOnCloning, dropEventsOnQueueFull, maxQueueSize are set for max reliability.

i.e., Is it possible that Light forwarder did send the data to forwarder but the forwarder had died before forwarding to indexer(s). At that point when forwarder comes back to life again can it recover from the data loss?

The above question might boil down to whether the forwarder's(listening on the network, not tailing a local file) queue is persistent or not?

We have a '0' dataloss requirement can it be achieved with this setup with proper configuration on Splunk?

Thanks help,

-Kris

[Stephen_Sorkin · Accepted solution]

There is currently no 0% data loss guarantee possible under all possible failure conditions, though we're actively working on that by adding an acknowledgment channel back to the forwarders.

Assuming only scheduled outages, as long as you have two indexers that you're sending to using autoLB lightweight forwarding, you can take down any system with no data loss.

Assuming you're reading from files, you can shut down forwarders at any time without loss, as we'll keep track of our location. For network inputs, a shut down forwarder cannot listen, so the implication is clear.

I'm not sure why you have a tier of forwarders here. Having that layer is typically detrimental to performance and manageability.

I wouldn't tune any of the parameters unless you have an elaborate deployment with cloning.

[kris2000]

Stephen,

  Thanks for your clear response. Appreciate it!!

Kris