Getting Data In

Reliability of a forwarder

kris2000
Explorer

Hello everyone ,

We are planning to have a Splunk setup as below:

LightForwarders -> Forwarders -> Indexers

Assuming that all the above instances are on different physical hosts -

When a forwarder goes down(either gracious or kill) is there a possibility that the data/events could be lost? This is assuming when all the values for parameters like blockOnCloning, dropEventsOnQueueFull, maxQueueSize are set for max reliability.

i.e., Is it possible that Light forwarder did send the data to forwarder but the forwarder had died before forwarding to indexer(s). At that point when forwarder comes back to life again can it recover from the data loss?

The above question might boil down to whether the forwarder's(listening on the network, not tailing a local file) queue is persistent or not?

We have a '0' dataloss requirement can it be achieved with this setup with proper configuration on Splunk?

Thanks help,

-Kris

1 Solution

Stephen_Sorkin
Splunk Employee
Splunk Employee

There is currently no 0% data loss guarantee possible under all possible failure conditions, though we're actively working on that by adding an acknowledgment channel back to the forwarders.

Assuming only scheduled outages, as long as you have two indexers that you're sending to using autoLB lightweight forwarding, you can take down any system with no data loss.

Assuming you're reading from files, you can shut down forwarders at any time without loss, as we'll keep track of our location. For network inputs, a shut down forwarder cannot listen, so the implication is clear.

I'm not sure why you have a tier of forwarders here. Having that layer is typically detrimental to performance and manageability.

I wouldn't tune any of the parameters unless you have an elaborate deployment with cloning.

View solution in original post

0 Karma

Stephen_Sorkin
Splunk Employee
Splunk Employee

There is currently no 0% data loss guarantee possible under all possible failure conditions, though we're actively working on that by adding an acknowledgment channel back to the forwarders.

Assuming only scheduled outages, as long as you have two indexers that you're sending to using autoLB lightweight forwarding, you can take down any system with no data loss.

Assuming you're reading from files, you can shut down forwarders at any time without loss, as we'll keep track of our location. For network inputs, a shut down forwarder cannot listen, so the implication is clear.

I'm not sure why you have a tier of forwarders here. Having that layer is typically detrimental to performance and manageability.

I wouldn't tune any of the parameters unless you have an elaborate deployment with cloning.

0 Karma

kris2000
Explorer

Stephen,

  Thanks for your clear response. Appreciate it!!

Kris

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...