All Apps and Add-ons

Forwarder reconnect time after quarantined connection

BryanBerry
Path Finder

If a given forwarder has quarantined a connection to due too many failed connection attempts, will the forwarder reattempt a connection after a span of time? If so, what is the reattempt interval for a quarantined connection?

By quarantine, I mean the below:

03-16-2012 14:20:24.904 +1000 INFO  TcpOutputProc - Connection to 10.1.2.3:9997 closed. Connection closed by server.
03-16-2012 14:20:24.904 +1000 WARN  TcpOutputProc - Applying quarantine to idx=10.1.2.3:9997 numberOfFailures=3

(Example logs pulled from another question, but I know this is what they look like)

1 Solution

bmacias84
Champion

Hello BryanBerry,

The answer is yes a forwarder will try to reconnect after it has been quarantined. The backoffOnFailure I believe is set to 310sec. There are two other settings in the outputs.conf that help control quarantine which are maxFailuresPerInterval and secsInFailureInterval.

You can override this by using autoLBFrequency in your tcp stanza.

Here one of my previous posts: Splunk-indexer-impact-to-splunk-forwarder-lost-connection-to-splunk-indexer

Also read section called Backoff Settings When Unable To Send Events to Indexer on the outpus.conf

Hope this helps or gets you started. Don't forget to vote up and/or accept answers if they help.

Cheers,

View solution in original post

bmacias84
Champion

Hello BryanBerry,

The answer is yes a forwarder will try to reconnect after it has been quarantined. The backoffOnFailure I believe is set to 310sec. There are two other settings in the outputs.conf that help control quarantine which are maxFailuresPerInterval and secsInFailureInterval.

You can override this by using autoLBFrequency in your tcp stanza.

Here one of my previous posts: Splunk-indexer-impact-to-splunk-forwarder-lost-connection-to-splunk-indexer

Also read section called Backoff Settings When Unable To Send Events to Indexer on the outpus.conf

Hope this helps or gets you started. Don't forget to vote up and/or accept answers if they help.

Cheers,

BryanBerry
Path Finder

Thanks piebob!

0 Karma

bmacias84
Champion

@BryanBerry,
Thanks for the correction.

0 Karma

piebob
Splunk Employee
Splunk Employee

BryanBerry: for future reference, the spec/example files in the documentation are populated directly from the same files that you see in the same version of Splunk, so you don't need to check to see if something is in the files on disk if it's not in the docs--it's the same file. that said, i'll make sure the docteam/dev knows that backoffOnFailure is missing from the spec. thank you!

0 Karma

BryanBerry
Path Finder

Checked out http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Outputsconf to confirm your findings. I don't see backoffOnFailure defined as a parameter - I only see it referenced by the secsInFailureInterval parameter.

I did just test manually; however, it appears it took 5 minutes and 10 seconds before attempting another connection.

If you change your 30sec to 300 sec / 5 minutes, I'll accept 😄

I also checked the outputs.conf in etc/system/default and say no reference to backoffOnFailure. Odd.

0 Karma

yannK
Splunk Employee
Splunk Employee

40 seconds if i remember.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...