Getting Data In

Applying quarantine and removing quarantine

bhsakarchourasi
Path Finder

Hi All,

This is kind of similar issue as mention on below link but since it was unanswered posting it again.
https://answers.splunk.com/answers/211112/applying-quarantine-removing-quarantine.html

We have installed Universal forwarder on a new server to send logs to Splunk cloud, since we didn't have direct connectivity to indexers we are sending logs to heavy forwarder and we didn't have connectivity to DS as well so we are doing manual configuration in /etc/system/local, but we are getting below errors in UF.

11-01-2019 16:37:21.594 +0800 INFO  TcpOutputProc - Removing quarantine from idx=xx.xx.xx.xx:9997
11-01-2019 16:37:21.780 +0800 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
11-01-2019 16:37:21.966 +0800 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
11-01-2019 16:37:21.967 +0800 WARN  TcpOutputProc - Applying quarantine to ip=xx.xx.xx.xx port=9997 _numberOfFailures=2
11-01-2019 16:37:22.142 +0800 INFO  DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
11-01-2019 16:37:25.062 +0800 WARN  TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group splunk has been blocked for 300 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

We have some other universal forwarders sending logs to the same heavy forwarder and it's working fine.

Thanks in advance any help will be much appreciated.

Thanks.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

The log entries you posted are expected behavior in a number of situations.
The top two are:
1. Incorrect pass4Symkey value between forwarder and cluster [general], or indexer master using indexer discovery.
2. Indexer cluster using indexer discover was cycled (rolling restart), and the forwarder has not received an updated list of available indexers. The timeout for discovery is configurable.

----
An upvote would be appreciated and Accept Solution if it helps!
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...