Getting Data In

Splunk Forwarder Pipeline Error

ifeldshteyn
Communicator

At random I am getting a strange heavy forwarder issue that no one seems to have received before (google comes up with nothing). Restarting the forwarder fixes the problem until it occurs again.

Whenever I log into my forwarder I get this message in the top right corner.

Tcp output pipeline blocked. Attempt '8300' to insert data failed.
12/6/2013 1:04:00 PM

(The frustrating thing is that, for some reason, the splunk logs do not show this error message. You can only see it when you log into the splunk forwarder interface.)

The number at the top increases as it seems to try and connect in a loop.

My outputs.conf (with redacted IP) on the forwarder are...

[tcpout]
autoLB = true
defaultGroup = default-autolb-group
indexAndForward = 0
maxQueueSize = 7MB

[tcpout:default-autolb-group]
disabled = 0
server = 123.123.123.123:9997
useACK=true

[tcpout-server://123.123.123.123:9997]

I've straced the splunk process and I see that it loops on fd 8 with EAGAIN error.

read(8, 0x7fff16023590, 256)            = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(3, {{EPOLLIN, {u32=369244720, u64=140733562632752}}}, 16, 4294967295) = 1
read(8, "P", 256)                       = 1
recvmsg(10, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {7}}, msg_flags=0}, 0) = 1
write(8, "Q", 1)                        = 1
write(8, "\0\0\0\0", 4)                 = 4
read(8, "F\200\222\r\1\0\0\0\0\200\226\r\1\0\0\0\0\220\222\r\1\0\0\0\0\220\0\0\0\0\0\0"..., 256) = 197
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b391dbacc50) = 23318
close(6)                                = 0
close(7)                                = 0
open("/splunk/var/run/splunk/splunkd.pid.tmp", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 6
write(6, "20618\n20619\n18020\n19104\n23318\n", 30) = 30
close(6)                                = 0
rename("/splunk/var/run/splunk/splunkd.pid.tmp", "/splunk/var/run/splunk/splunkd.pid") = 0
write(8, "O", 1)                        = 1
write(8, "\20\0\0\0\0\0\0\0", 😎        = 8
read(8, 0x7fff16023590, 256)            = -1 EAGAIN (Resource temporarily unavailable)
epoll_wait(3, 2b391e0fda80, 16, 4294967295) = -1 EINTR (Interrupted system call)
--- SIGCHLD (Child exited) @ 0 (0) ---

In Proc it shows that FD 8 -> socket:[26992686] and netstat says it is Connected and the tcp socket is in a CLOSE_WAIT mode.

Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  3      [ ]         STREAM     CONNECTED     26992686

Proto Recv-Q Send-Q Local Address               Foreign Address             State
tcp   1      0      FWDHOST:40537               IDXHOST:palace-6            CLOSE_WAIT

I see no errors on the indexer side and I am assuming the problem is getting that splunk is not attempting to restart the connection that has clearly hung. I am not sure if a difference in the versions of the forwarder vs indexer is an isssue (6 vs 5.05).

Anyone have any points of wisdom of what I can do to resolve this (other than restarting).

Thanks,
Ilya

1 Solution

ifeldshteyn
Communicator

So after much debugging I realized that the new 6+ versions seem to have this error. When I downgraded Splunk Forwarder and Indexer to 5.0.5 this error went away. I hope this helps others who were stuck with me.

View solution in original post

0 Karma

jeffrogers
Explorer

I had this problem on a heavy forwarder taking things in with syslog-ng. Very simple test setup at this point only a couple files. My outputs.conf looked like the one in the original post except I was using hostnames not IP's. Turns out DNS was not resolving correctly, needed the FQDN or a DNS Search Path and it started working.

I'm assuming this particular error message may be a catch-all for general communication problems.

0 Karma

ifeldshteyn
Communicator

So after much debugging I realized that the new 6+ versions seem to have this error. When I downgraded Splunk Forwarder and Indexer to 5.0.5 this error went away. I hope this helps others who were stuck with me.

0 Karma

nikhilagrawal
Path Finder

Hi I have recently upgraded to Splunk version6. I am facing the same issue.

Tcp output pipeline blocked. Attempt '200' to insert data failed.

Can someone suggest solution please?

0 Karma

ifeldshteyn
Communicator

I turned on DEBUG mode but I only see the pipeline blocked message in the webservice log (when I open the webpage). I cut out some of the garbage. I still see nothing that would id this issue in logs without opening page.

2013-12-11 16:48:41,595 DEBUG [52a8ddb9967083c90] proxy:357 - [Splunkweb Proxy Traffic] response body:

{[{"name":"TCPOUT_SEND_FAILED","id":"https://123.123.123.123:8089/services/messages/TCPOUT_SEND_FAILED","updated":"2013-12-11T16:48:41-05...
TCPOUT_SEND_FAILED","remove":"/services/messages/TCPOUT_SEND_FAILED"}

0 Karma

bcauer
New Member

I'm having the same issue. Any solution to this...............

0 Karma

Gilgalidd
Path Finder

Check your configuration files (etc/system/local/*).For this time, the message don't appear since I have do some change (don't ask me, I don't remember the changes I've made, inputs/props/trandforms) on Universal and Heavy.

Don't miss to share the solution if you have it.

0 Karma

ifeldshteyn
Communicator

Thank you, at least now I know that I am not the only poor soul that has this issue. It seems to be extremely rare because no one mentions it anywhere (google, answers etc...) . Something is making Splunk forwarder connection hang. Maybe it gets blocked by the indexer and the forwarder does not know it? In any case, there are no errors on the indexer-side. It just stops receiving one moment...

Thanks for confirming that this is not a versioning problem. We may have to reach out to Splunk enterprise support.

Gilgalidd
Path Finder

Hello,

I am not sure if a difference in the versions of the forwarder vs indexer is an isssue (6 vs 5.05).

Apparently not, I've the same issue with the following topology :

Universal Forwarder (A) >> Heavy Forwarder (B) >> Indexer (C)

A : splunkforwarder-6.0.1-189883-x86-release.msi

B : splunk-6.0.1-189883-x86-release.msi

C : Splunk Version...6.0S - Splunk Build...182037

And I see this message only on the Heavy Forwarder (B).

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...