Too much Lag in CDC Pipeline (Confluent JDBC Sink Connector)

Kafka Connect
1

I am currently managing a CDC (Change Data Capture) Pipeline for data streaming, where the source database is PostgreSQL and using the Debezium Source Connector is streaming data from Source PostgreSQL database to Kafka Topic with publishing rate of 4.5 messages/second. The data is streamed to two separate PostgreSQL databases as the target, facilitated by Confluent JDBC Sink Connectors (Version 10.6.4) that read data from Kafka.

I am facing an issue with a persistent lag of approximately 45 messages in one of the Confluent JDBC Sink Connectors. Strikingly, the other Confluent JDBC Sink Connector, which feeds data into a different PostgreSQL database, experiences a lag of only 15 to 25 messages.

Despite my attempts to reduce this lag i tried to modify the following properties:

consumer.override.max.poll.records=1000
max.poll.interval.ms=180000

consumer.override.fetch.min.bytes = 101200
consumer.override.fetch.max.wait.ms = 1000
consumer.override.auto.commit.interval.ms=5
consumer.override.offset.flush.interval.ms=100
batch.size=4000

The lag has not reduced significantly. I am seeking suggestions on how to address this issue effectively and reduce the lag in the affected Confluent JDBC Sink Connector.

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.