Getting Data In

Forwarder resiliency

mundus
Path Finder

My understanding was that when a forwarder loses its connection to the central Splunk server, it will continue accepting data from other forwarders and it will queue those events until the Splunk server comes back online. One of our engineers is under the impression that if the central Splunk server is unavailable, forwarders stop accepting new data or just drop it on the floor.

Could somebody clarify this for me?

Thx.

Craig

Tags (1)

_d_
Splunk Employee
Splunk Employee

You're partially right; if an intermediate forwarder loses its connection to an indexer it will accept data from other forwarders until its own queues get filled up and then it will start dropping them.

This is true even when an indexer's queues are filled up (ie. the problem propagates down the tiers all the way to the first level forwarder). This is precisely why our preferred, most reliable and recommended method of data input is to first persist a log file to disk then monitor it using a Universal Forwarder. It is capable - among other things - of keeping track of what parts of a file have been sent to the indexer therefore guaranteeing you to have all your data safely send to the indexer in case something happens upstream (the connection is dropped, parts of the network go down, a firewall comes up, the indexer goes down for an upgrade, an indexer's queues are filled up etc). Hope it clarifies the problem a bit.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...