We have implemented Debezium with kafka and Schema Registry with AVRO, we are experience a performance issue when both, source and sink connectors, are working at the same time the consumer performance decrease significantly.
All the setup is running in an Azure Kubernet Cluster AKS
Here some number of what we have:
This is an standart kafka cluster deployed using strimzi operator: https://strimzi.io/
1 Kafka Cluster with 3 nodes (8GB ram -Xms: 4096m -Xmx: 4096m each) - All the topics with Replication Factor = 3
1 Zookeper Cluster with 3 nodes (4GB RAM -Xms: 2048m -Xmx: 2048m each)
1 Connect Cluster with 5 nodes (12GB RAM c/u - -Xms: 6168m -Xmx: 6168m) with 1 Partition and 1 task but we have tried with 1 / 5 / 100 partitions and 1 / 5 / 15 / 100 task
1 Source Connector per table (4 tables) - io.debezium.connector.sqlserver.SqlServerConnector
1 Sink Connector per Table (4 tables) - io.confluent.connect.jdbc.JdbcSinkConnector
The problem: the Target DB has a delay that increase when the quantity of messages is more than 750msjs per seconds.
Mesage size = 6kb
Now we are processing 2000 msjs of 6kb per second (This was the best performance that we get and it was enabling ONLY the sink connectors)
When we enabled only the sink connectors the performance is good, the same with source conectors with no sinks, performance is great
Resources are OK, source and target environments have pleanty of CPU and Memory.
When we see the messages consume per second there is like a wait for 30sec and then it consume some msjs and wait and then again.
We tested kafka (without connectors) using the following scripts: bin/kafka-producer-perf-test.sh - bin/kafka-consumer-perf-test.sh that are included in the kafka software and the performance was pretty good (1Mill per Second):
What we need at this moment is advice about what we need to change to improve the performance of the connectors significantly.
Is there anyone who can help us? Any kind of support that we can adquire?