Getting Data In

Forwarder resiliency

mundus
Path Finder

My understanding was that when a forwarder loses its connection to the central Splunk server, it will continue accepting data from other forwarders and it will queue those events until the Splunk server comes back online. One of our engineers is under the impression that if the central Splunk server is unavailable, forwarders stop accepting new data or just drop it on the floor.

Could somebody clarify this for me?

Thx.

Craig

Tags (1)

_d_
Splunk Employee
Splunk Employee

You're partially right; if an intermediate forwarder loses its connection to an indexer it will accept data from other forwarders until its own queues get filled up and then it will start dropping them.

This is true even when an indexer's queues are filled up (ie. the problem propagates down the tiers all the way to the first level forwarder). This is precisely why our preferred, most reliable and recommended method of data input is to first persist a log file to disk then monitor it using a Universal Forwarder. It is capable - among other things - of keeping track of what parts of a file have been sent to the indexer therefore guaranteeing you to have all your data safely send to the indexer in case something happens upstream (the connection is dropped, parts of the network go down, a firewall comes up, the indexer goes down for an upgrade, an indexer's queues are filled up etc). Hope it clarifies the problem a bit.

Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...