All Apps and Add-ons

Splunk Add-on for Kafka: Why is there low performance after configuration?

wangjiaji
Engager

I'm using Splunk Add-on for Kafka to collect data from a Kafka cluster, I believe the configs are OK because data is coming, but the transmission speed is very low, about 200 msg/min (the transmission speed into Kafka is about 1000 msg/min), but once in an hour or two, there are 20k msg/min for 1-2 minutes. The Kafka cluster is on the same machine, so I think network won't be a problem. I have repeated the same configuration on 3 different machines, and the same thing happens, what could've caused this issue?

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

This page refers to batch mode... https://www.splunk.com/blog/2015/10/12/achieving-scale-with-the-kafka-modular-input/

Enable Batch Mode. This will buffer events in memory until the batch buffer is flushed depending on how you tune the flush settings.You can tune the size of the batch buffer depending on the scale of the throughput in your Kafka environment ie: higher throughput => larger batch buffer will be more optimal.

Sounds to me like you want a smaller batch size.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...