Getting Data In

forwarder stopped sending to indexer but continues to send to 3rd party receiver

Log_wrangler
Builder

Any help on this is greatly appreciated.

I have a bunch of servers with UFs sending to a HF that is configured to send cooked data to splunk indexers and raw uncooked to a 3rd party receiver.

The flows have been running fine, but suddenly data to the indexers stopped.

I am looking for any suggestions to troubleshoot this.

Thank you

0 Karma
1 Solution

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

View solution in original post

0 Karma

Log_wrangler
Builder
  • should mention, that eventually everything stopped receiving, after a few minutes.
0 Karma

Log_wrangler
Builder

Thanks, did that and could see that the 3rd party receiving ip was blocking, so there must have been a build up that ultimately stopped the forwarder...

02-22-2018 16:44:56.119 -0600 INFO TcpOutputProc - Connected to idx=1.1.1.1:some_port
02-22-2018 16:53:11.041 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 10 seconds.
02-22-2018 16:53:21.024 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 20 seconds.
02-22-2018 16:54:09.009 -0600 WARN TcpOutputProc - Forwarding to indexer group Subsidiary blocked for 40 seconds.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Check the logs.

0 Karma
Get Updates on the Splunk Community!

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...