Getting Data In

Splunk stopped receiving forwarder connections

MHibbin
Influencer

Hey,

Just wondered if anyone has seen this issue in their environment?

I noticed, by chance, that our license usage was particularly low for the day so far. I soon found that most of the forwarders had stopped forwarding traffic to our Indexer.

I found the following messages in splunkd.log:

08-20-2013 04:55:38.976 +0100 INFO  TcpInputProc - Stopping IPv4 port 9997
08-20-2013 04:55:38.976 +0100 WARN  TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds
08-20-2013 04:55:41.004 +0100 INFO  TcpInputProc - Starting IPv4 port 9997
08-20-2013 04:55:41.004 +0100 WARN  TcpInputProc - Started listening on tcp ports. Queues unblocked

Although the messages imply Splunk started listening again 9997, it appears this is not entirely correct, as we lost a lot of of data for about 10 hours, luckily some of which can be retrieved, however some of it cannot. Restart Splunk seems to have fixed this issues, well allowing traffic on 9997 at least. Before restarting Splunk had ownership of 9997.

Cheers,

Matt

0 Karma

BP9906
Builder

lukejadamec is correct. I ran into this a long time ago on my first splunk install and setup. I also found Linux to run splunk indexers more efficiently.

0 Karma

lukejadamec
Super Champion

I have seen all connections to forwards drop a number of times for no apparent reason. Adding this to the system//inputs.conf file on the indexer seems to have fixed the problem.
[splunktcp:9997]
Connection_host=none

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...