Getting Data In

How does Splunk Universal Forwarder behave for load balanced deployment toplogies when Receivers are down?

koshyk
Super Champion

one of the customers have a situation whereby there are 1000's of clients with Universal Forwarders in multiple network zones , trying to reach Splunk Heavy Forwarders which are also in multiple network zones. The network zones has to be specific due to security controls, but it is very hard to determine which zone the client (UF) beforehand. As of now, the outputs.conf are hand-crafted manually once the customer identifies which zone the UF is based upon.
I was thinking to push outputs.conf with All Heavy-forwarder-servers in outputs.conf, but I'm sure some of these cannot be reached from the clients. So my question is

  1. How does the UF load-balance behave when it has all (say 10) servers in its outputs.conf list, but only can reach a subset (say 4) of them?
  2. Will it throw error and cause failure on the client? or lot of error logs?
  3. Is there mechanism whereby we can ask the UF not to try the receiver again if it fails N number of times?
0 Karma
1 Solution

woodcock
Esteemed Legend

It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.

View solution in original post

woodcock
Esteemed Legend

It will generate timeout logs and then move on to the next indexer. The built-in load-balancing does not provide a way to automatically stop trying an Indexer that is continuously down.

koshyk
Super Champion

I hope that means, all the data will be intact but will have errors in the UF logs?

0 Karma

woodcock
Esteemed Legend

No data loss, but possibly data duplication (very unlikely), unless you useAck.

0 Karma

ridwanahmed
Path Finder

Is this the same for if an indexer has full disk?

0 Karma

woodcock
Esteemed Legend

Yes, the indexer should put itself into detention/quarantine.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

This documentation page has everything you need to answer you own question.

https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Protectagainstlossofin-flightdata

koshyk
Super Champion

I did read this before posting. The actual statement, i wanted to understand from that document was

In all these cases, the forwarder will then attempt to open a connection to the next indexer in the load-balanced group, or to the same indexer again if load-balancing is not enabled.

But I'm not sure whats the impact of having continous non-reachable/timeout indexers

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...