Security

What caused so many UNREPLIED dport=9997 ip_conntrack entries?

rkilen
Explorer

Running Splunk 5.0.5 on RHEL 5.11.

One of our Splunk servers recently had several messages appear in dmesg like this:
ip_conntrack: table full, dropping packet

A bit of research led to the net.ipv4.ip_conntrack_max setting, which we doubled to eliminate the messages while we investigated. Found a large number of "UNREPLIED" tcp entries in /proc/net/ip_conntrack which had long time to live (up to five days), and had dport=9997, so were destined for the indexer. These entries were the vast majority of the ip_conntrack table entries.

We looked at the rest of our Splunk servers, and found one other where the ip_conntrack table was nearly full, but most of the indexers' ip_conntrack tables were closer to 10% of capacity, most of which were UNREPLIED entries for dport=9997.

We're trying to find out what is causing these UNREPLIED entries, but have so far been unsuccessful. We've also found conflicting information on the web as to whether "dropping packet" means that a real IP packet was dropped by iptables, or merely that an ip_conntrack table entry was dropped to make room for a NEW packet.

Has anyone else seen this, or have suggestions about the cause and fix?

Tags (3)
0 Karma

elvisior
Explorer

Hi, I'm seeing the same issues on some rhel5 boxes that have iptables enabled.

Did you ever find a solution to this?

The bandaids I've seen suggested elsewhere are to increase net.ipv4.netfilter.ip_conntrack_max.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...