Getting Data In

Splunk stopped receiving forwarder connections

MHibbin
Influencer

Hey,

Just wondered if anyone has seen this issue in their environment?

I noticed, by chance, that our license usage was particularly low for the day so far. I soon found that most of the forwarders had stopped forwarding traffic to our Indexer.

I found the following messages in splunkd.log:

08-20-2013 04:55:38.976 +0100 INFO  TcpInputProc - Stopping IPv4 port 9997
08-20-2013 04:55:38.976 +0100 WARN  TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds
08-20-2013 04:55:41.004 +0100 INFO  TcpInputProc - Starting IPv4 port 9997
08-20-2013 04:55:41.004 +0100 WARN  TcpInputProc - Started listening on tcp ports. Queues unblocked

Although the messages imply Splunk started listening again 9997, it appears this is not entirely correct, as we lost a lot of of data for about 10 hours, luckily some of which can be retrieved, however some of it cannot. Restart Splunk seems to have fixed this issues, well allowing traffic on 9997 at least. Before restarting Splunk had ownership of 9997.

Cheers,

Matt

0 Karma

BP9906
Builder

lukejadamec is correct. I ran into this a long time ago on my first splunk install and setup. I also found Linux to run splunk indexers more efficiently.

0 Karma

lukejadamec
Super Champion

I have seen all connections to forwards drop a number of times for no apparent reason. Adding this to the system//inputs.conf file on the indexer seems to have fixed the problem.
[splunktcp:9997]
Connection_host=none

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...