I am observing packet loss on Heavy forwarder due to which I am missing the important messages which we are being sent using snmp traps. I have already increased the rmem buffer size to the suggested value for splunk stream app on Splunk docs(which I thought should be more than enough) , but even after that change there are still a lot of packet drops on the HF.
current stats:
sysctl net.core.rmem_max
net.core.rmem_max = 33554432
netstats:
netstat -suna
Udp:
52071486 packets received
21017 packets to unknown port received.
3747277 packet receive errors
82100 packets sent
3747277 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 27
InMcastPkts: 8
InOctets: 31643507863
OutOctets: 6061193400
InMcastOctets: 288
InNoECTPkts: 62078913
InECT0Pkts: 1301
Any idea, what should be the ideal size for the net.core.rmem_max that can guarantee receive buffer errors reduce to zero.
Or this is something which we cannot achieve by increase the buffer size?
Based on your HF hardware capacity, set one of the below for the UDP input that you've:
queueSize = <integer>[KB|MB|GB]
* Maximum size of the in-memory input queue.
* Default: 500KB.
persistentQueueSize = <integer>[KB|MB|GB|TB]
* Maximum size of the persistent queue file.
* Persistent queues can help prevent loss of transient data. For information on
persistent queues and how the 'queueSize' and 'persistentQueueSize' settings
interact, search the online documentation for "persistent queues"..
* If you set this to a value other than 0, then 'persistentQueueSize' must
be larger than either the in-memory queue size (as defined by the 'queueSize'
setting in inputs.conf or 'maxSize' settings in [queue] stanzas in
server.conf).
* Default: 0 (no persistent queue).
In addition I suggest to use two Heavy forwarders with a Load balancer to distribute load and be sure of HA features!
Bye.
Giuseppe
Have you tried enabling useACK=true
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf