Hey All, Is the S3 Sink connector looking for a specific key in the Kafka message to extract data? When looking at my message in Kafka I see the following keys in the “top” level: “schema” and “payload”, but when I look at the files (“parquet” format) in S3, I only see the “payload” data. Is the c…

It sounds like you’re using org.apache.kafka.connect.json.JsonConverter with schemas.enable=true. When set, the schema is enabled within the JSON message. Only the payload itself is sent to S3. See Kafka Connect Deep Dive – Converters and Serialization Explained | Confluent

Yes. But you might want to consider using Avro/Protobuf/JSON Schema so that the schema is still retained and available for other consumers of the data.

Hey, I have tried to add -e CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE="false" \ -e CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE="false" to my kafka connect container, to remove the schema from debezium event and that worked, but not I’m getting the following error: {"id":0,"state":"FAILED","worker_id":…

How the S3 Sink connector extract data

Kafka Connect

shlomi 30 March 2021 15:20 3

Thanks, so I should be able to remove it, by setting value.converter.schemas.enable=false in the source connector, and it should not have any effect on the S3 sink. right?

Topic		Replies	Views
Getting below exception in kafka s3 sink connector Kafka Connect	1	388	6 August 2024
S3 Sink Connector only contains part of Kafka topic Kafka Connect	5	3901	14 February 2022
KafkaConnect with Amazon Sink S3 Sink Connect is not working Self-Managed Connectors	12	12277	2 July 2022
Help with error while using amazon S3 sink connector Self-Managed Connectors	1	281	13 March 2025
Configuring s3 connector using custom schema registry Self-Managed Connectors	0	29	21 September 2024

How the S3 Sink connector extract data

Related topics