Using Multiple Connectors VS Multiple Topics on Single Connector

I have two Kafka topics (topic_A and topic_B) with identical partitioning (4 partitions) and similar data. Both topics require the same processing work.

Currently, I’m considering a single Kafka Connect Sink connector to write data to BigQuery, with tasks.max set to 8. This setup will run within a Kubernetes (K8s) cluster configured to scale up pods/workers based on topic latency, pod CPU, or pod memory usage.

Are there advantages to using two separate Sink connectors (one per topic) instead of a single connector for this scenario? If so, what are they? I’m primarily interested in factors like performance, fault tolerance, scaling flexibility, and overall management.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.