- What is the ideal number of workers that should be configured in the connector? How is this number decided?
- Can I horizontally scale the connector? Or can it run only on one machine?
There is no “ideal” number. You’ll need to benchmark your own hardware + processes.
Yes, you can deploy a connect cluster on many machines, as mentioned in the Kafka and Confluent documentation. max.tasks in the connector config will scale the process to all workers in the cluster. Keep in mind that some connectors cannot have more than one task - if you have one topic partition, or must be one process, such as writing to local-disk or a database table with write locks
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.