I am currently managing a CDC (Change Data Capture) Pipeline for data streaming, where the source database is PostgreSQL and using the Debezium Source Connector is streaming data from Source PostgreSQL database to Kafka Topic with publishing rate of 4.5 messages/second. The data is streamed to two separate PostgreSQL databases as the target, facilitated by Confluent JDBC Sink Connectors (Version 10.6.4) that read data from Kafka.
I am facing an issue with a persistent lag of approximately 45 messages in one of the Confluent JDBC Sink Connectors. Strikingly, the other Confluent JDBC Sink Connector, which feeds data into a different PostgreSQL database, experiences a lag of only 15 to 25 messages.
Despite my attempts to reduce this lag i tried to modify the following properties:
consumer.override.max.poll.records=1000 max.poll.interval.ms=180000 consumer.override.fetch.min.bytes = 101200 consumer.override.fetch.max.wait.ms = 1000 consumer.override.auto.commit.interval.ms=5 consumer.override.offset.flush.interval.ms=100 batch.size=4000
The lag has not reduced significantly. I am seeking suggestions on how to address this issue effectively and reduce the lag in the affected Confluent JDBC Sink Connector.