How come my forwarders appear frozen and are unable to send data to the indexers?

[sylim_splunk]

We have a lot of forwarders, of universal forwarders and heavy forwarders mostly, in version 7.0.x, which are configured to send data to 10s of indexers.

Intermittently, some forwarders fail to send data putting a lot of messages in the splunkd.log then it becomes fine after a restart.

Around the time of incident, one of 10s indexers had disk issue and had connectivity issue.

======[ Splunkd.log ]==
WARN TcpOutputFd - Connect to 11.22.33.44:9997 failed. Connection refused
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 10 seconds.
[ snip ]
WARN TcpOutputProc - Tcpout Processor: The TCP output processor has paused the data flow. Forwarding to output group default-autolb-group has been blocked for 5790 seconds.
==============================

[sylim_splunk · Accepted solution]

We have seen this issues from time to time and have been working on it to get to the bottom of issue.
This can be worked around by using forcedTimebasedAutoLB = true until the fix is found and released.

[juancamiloll]

in which path and file should I enter that value? @sylim_splunk 

[youngsuh]

@juancamiloll 

Time based load balancing | Splunk

The answer you seek is output.conf