Parallelism and Load Balancing in Distributed Kafka Connect Deployment

Kafka Connect
1

I have two questions regarding Kafka Connect in a distributed deployment model with multiple workers:

  1. How are tasks load-balanced across the workers?

    • I understand that for each connector, we can configure a specific number of tasks, including a maximum number of tasks. What algorithm is used to distribute these tasks among the workers to ensure an equal load? Does the algorithm take resource utilization into account?

  2. How many tasks can be run in parallel on a single worker? Does this number change if the tasks come from different connectors?

    • From my understanding, the load is balanced across workers based on the number of tasks. How is the number of tasks assigned to each worker determined? Is it always one task per worker at a given point of time, with additional tasks queued until the current ones are completed?
2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.