I'm using the Splunk Add-on for Kafka to collect events from a kafka topic. The kafka server was on the same machine with splunk, so network should not be an issue.
There is a shell script continuously populating events into a certain kafka topic with 16 partitions (let's say topic "A"), when I use kafka-console-consumer.sh to check topic "A", it prints events continuously which is just as expected. So I thought my kafka setup was correct.
Now the problem is,
When I was populating the events into the topic "A" in normal speed (50-80 events per minute), the splunk add-on for Kafka fetches messages in a strange behavior, it got hundreds of events in 1 minute but "sleep" for another 6-7 minutes. Just like the chart below,
But, when I increased the populating speed to 1000 msg/s or 5000 msg/s, splunk got those messages immediately and in expected throughput, please see the screenshots below,
It seems like there is some "buffer" in the add-on which will wait for certain count of messages to flush into splunk...
btw, the Kafka version is:
kafka_2.12-0.10.2.0
the inputs.conf content:
[kafka_mod]
interval = 5
[kafka_mod://kfk1]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 0
[kafka_mod://kfk2]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 1
[kafka_mod://kfk3]
index = aaa
kafka_cluster = bbb
kafka_partition_offset = earliest
kafka_topic = A
kafka_topic_group = splunk
kafka_partition = 2
...
//16 partitions in total
...
The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.
The problem was solved, my colleague found some parameters controlling the batch behavior in the add-on.
How do we get Kafka Key Value pair in Splunk. Always we see only Kafka Value, we need Kafka Key as well
I have the same problem: I have a script writing 1 event per second on a Kafka environment but the Splunk Add-on for Kafka collects more or less 60 events on spikes every one minute.
How did you end up solving the issue?
I just tested with kafka_2.10-0.8.2.2, the same result.