All Apps and Add-ons

Splunk Add-on for Kafka: spikes with very long gaps on low throughput

sonicant
Path Finder

I'm using the Splunk Add-on for Kafka to collect events from a kafka topic. The kafka server was on the same machine with splunk, so network should not be an issue.
There is a shell script continuously populating events into a certain kafka topic with 16 partitions (let's say topic "A"), when I use kafka-console-consumer.sh to check topic "A", it prints events continuously which is just as expected. So I thought my kafka setup was correct.

Now the problem is,
When I was populating the events into the topic "A" in normal speed (50-80 events per minute), the splunk add-on for Kafka fetches messages in a strange behavior, it got hundreds of events in 1 minute but "sleep" for another 6-7 minutes. Just like the chart below,
alt text

But, when I increased the populating speed to 1000 msg/s or 5000 msg/s, splunk got those messages immediately and in expected throughput, please see the screenshots below,

alt text

It seems like there is some "buffer" in the add-on which will wait for certain count of messages to flush into splunk...

btw, the Kafka version is:
kafka_2.12-0.10.2.0

the inputs.conf content:

[kafka_mod]
interval = 5

[kafka_mod://kfk1]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 0

[kafka_mod://kfk2]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 1

[kafka_mod://kfk3]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 2

...
//16 partitions in total
...
1 Solution

sonicant
Path Finder

The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.

View solution in original post

sonicant
Path Finder

The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.

sivakumarsamy
New Member

How do we get Kafka Key Value pair in Splunk. Always we see only Kafka Value, we need Kafka Key as well

0 Karma

davidepiotti
Explorer

I have the same problem: I have a script writing 1 event per second on a Kafka environment but the Splunk Add-on for Kafka collects more or less 60 events on spikes every one minute.
How did you end up solving the issue?

0 Karma

sonicant
Path Finder

I just tested with kafka_2.10-0.8.2.2, the same result.

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...