I have two Kafka topics (topic_A and topic_B) with identical partitioning (4 partitions) and similar data. Both topics require the same processing work.
Currently, I’m considering a single Kafka Connect Sink connector to write data to BigQuery, with tasks.max set to 8. This setup will run within a Kubernetes (K8s) cluster configured to scale up pods/workers based on topic latency, pod CPU, or pod memory usage.
Are there advantages to using two separate Sink connectors (one per topic) instead of a single connector for this scenario? If so, what are they? I’m primarily interested in factors like performance, fault tolerance, scaling flexibility, and overall management.