Out of Memory Issue
Hi All,
I’m new to Kafka Connect and need some guidance regarding an issue we’re encountering.
We are managing 500+instances of source Postgres (9.3)(one database per tenant) in our legacy system, each with 14 tables that we need to listen for updates.
Currently, we’ve created one connector per table. However, we plan to consolidate this into one connector per database moving forward.
We are using the official Confluent Docker image: confluentinc/cp-kafka-connect-jdbc, with the following heap settings:
-e KAFKA_HEAP_OPTS="-Xms4g -Xmx6g"
The setup includes a minimum of 10 instances and a maximum of 15 instances, with a batch size of 100 and a polling interval of 1 minute.
Despite these settings, the application runs out of memory over time, and we are facing memory issues.
Business Context: We do not receive more than 1000 records per connector per minute. Our approach is to start with batch mode and then update the connector to use the timestamp+incrementing mode.
After analyzing the memory dump, I observed that:
- A
byte[]object is taking up ~550MB of memory. - A
HashMapis consuming more than 2GB, and it is holding many connector objects.
I’ve attached the heap dump screenshot showing these details.
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “kafka-producer-network-thread | connector-producer-C149429_AOMD_po_invoice_detail_Reconcile_19_11_2024_PO-0”
