I am working on creating a mysql CDC pipeline using debezium connector and the sink connects to HDFS using hdfs-sink connector, everything was fine until my source connector failed due to record too large exception, now i know i need to change the max.request.size to a value more than that, but i am not sure where do i do that, i have made the changes in distributed.properties but it did not work for me, please help me where does confluent pick the producer config.
{"name":"exact-test","connector":{"state":"RUNNING","worker_id":"x.x.x.x.:8083"},"tasks":[{"id":0,"state":"FAILED","worker_id":"x.x.x.x:8083","trace":"org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:282)
at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:336)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:264)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1522947 bytes when serialized which is larger than 1048576,
To override producer values in Kafka Connect use the producer. prefix in the Kafka Connect worker config (i.e. producer.max.request.size). You can also override it per connector.
Keep in mind that if you change the producer max.request.size setting, you will need to change the corresponding topic max.message.bytes setting. You may also need to change the consumer fetch.max.bytes and max.partition.fetch.bytes settings. Here are links to the doc for these settings.
HTTP/1.1 400 Bad Request
Date: Wed, 07 Apr 2021 15:03:48 GMT
Content-Type: application/json
Content-Length: 319
Server: Jetty(9.4.24.v20191120)
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nThe 'None' policy does not allow 'max.request.size' to be overridden in the connector configuration.\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
Please anyone can tell me which file should i change the below parameters.
connector.client.config.override.policy=All
I am extremely sorry about this, but I am aware about the property but I am still not able to understand which file is the worker file here.
Here I am registering the my sql source connector directly using a curl command and hence not using a file to do it.
Here under $CONFLUENT_HOME/etc/kafka I see couple of files is it any one of these? #producer.properties #server.properties #consumer.properties #connect-distributed.properties #connect-standalone.properties
Even I have tried adding the property in all the three files (producer, server and consumer) files
Just reading through this and was wondering if you ever got to the bottom of it?
I’m currently having a similar issue, whereby I want to increase the max request size to allow bigger batches through from Debezium to fix a bottleneck in out system. I’ve updated my settings as described above and I’ve set the cluster and connector settings but the metric kafka_connect_connect_metrics_request_size_max doesn’t appear to increase. To test if I get the same value before the connector is applied I tried removing it and still got the same value for this metric which is 99 but I’m not sure what that really means, is it measured in KB or MB?
We’re running kafka connect in kubernetes using the docker image confluentinc/cp-kafka-connect:6.2.1. When I delete the pods and they are recreated the metric above starts at an increased size and then caps off at the same value every time. I can’t find the source code that shows how this metric is set so I’m unsure what might be causing it but in my attempts to fix the issue I’ve changed the following:
Kafka Connect Config:
Hopefully I’m not falling down a bit of a rabbit hole here as I’m assuming kafka_connect_connect_metrics_request_size_max is the absolute maximum size of a producer message that can be sent from the connectors in kafka connect? It would be nice to see exactly what sets this metric so any help would be appreciated.
I only set the variable in the Kubernetes manifests:
CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY: All
I restarted the connect pods and then I set the configuration of the connector in Control Center, but it does not pick up the changes. The Kafka Connect connector ignores all the options producer.override or consumer.override according to the logs.