Hi Team,
We have received the following complaint from a customer using a Kafka Streams application deployed with version 3.3.1:
"We are noticing a CDR lag in the generated files, which needs to be analyzed to determine the cause. This issue is impacting management reporting and supporting customer complaints. Your assistance is required. Customers have specific queries and are seeking answers on a bridge call. Here is an example of one such query:
- What is the relation between 216 partitions and corresponding 216 tasks, and the limitation of writing only 60 files at the same time? How does the partition decide which file to write to out of those 60 files?”
Customer’s Full List of Questions:
- What is the relation between 216 partitions and 216 tasks, and the limitation of writing only 60 files at the same time?
- How does the partition decide which file to write out of those 60 files?
- What is the backlog between the partitions?
- How does Kafka operate between partitions, tasks, limitations, and record writing?
- Kafka Connect does not use round-robin partition assignment by default. How does it decide which partition to consume from?
IBM Server Team Response:
The IBM server team clarified that these questions relate to the client-side setup and that the team who developed the client application would be best suited to provide answers. They also mentioned that the queries pertain to the Kafka Connect client library.
Development Team Feedback:
Our development team confirmed that once partitions are assigned to a task, they are consumed like any other Kafka consumer. The documentation mentions that polling fetches messages from all assigned partitions but does not specify whether the consumer prioritizes partitions with more records or uses any priority mechanism. They advised that IBM support for Kafka Client or Kafka Connect would be best suited to explain the exact algorithm.
Assistance Requested:
We need the Kafka client/connect teams to provide insights on the following queries:
- What is the relation between 216 partitions, 216 tasks, and the limitation of writing only 60 files at the same time?
- How does the partition decide which file to write out of those 60 files?
- What is the backlog between partitions?
- How does Kafka operate between partitions, tasks, limitations, and record writing?
- How does Kafka Connect decide which partition to consume from, given it doesn’t use round-robin partition assignment by default?
We specifically require clarity on the exact algorithm used to determine whether the consumer prioritizes partitions with more records or applies any priority mechanism.
Your prompt assistance on this matter would be greatly appreciated.