How to enable compression in Kafka Source Connector (JDBC Source Connector)

Kafka Connect
1

I tried using this code to enable compression in the source connector. It is not working. Also my cluster is enabled with auto.create.topics.enable = true.

curl -H 'Content-Type:application/json' -X POST localhost:8083/connectors -d ' {
"name": "Kafka_Compression_Connector",
"config": {
        "connector.class"                          : "io.confluent.connect.jdbc.JdbcSourceConnector",
		"connection.password"					   : "pwd",
		"topic.prefix"							   : "kafka_Compression_test"
		"mode"                                     : "bulk"
		"table.whitelist"                          : "testtable"
		"poll.interval.ms"                         : "3000"
		"topic.creation.default.partitions"        : 4,
        "topic.creation.default.replication.factor": 1,
        "topic.creation.default.compression.type"  : "snappy"
        }'

When I tested with 10000 records from the table. Size of the topic with compression and without compression is same. On describing the topic in cluster. Compression Type is not showing.

2

Hi @Poornaraj , is the topic already created? That config will be used only on topic creation

3

Hi, No the topic is not created. It will be created once the Source Connector is running, as we have turned on the auto.create.topics.enable=true.

4

Just because auto.create.topics.enable=true doesn’t mean that the topic might not already exist, e.g. from a previous test.

Can you try the test again, and show the output here of topic list beforehand, Kafka Connect config used, and then config properties list afterwards of the created topic?

5

Hi Sir, I could see the topic has been created automatically using the topic_prefix_name with the table name after that. But for 10K records it has occupied 10MB without any compression enabled in the source connector. After i added below lines to the source code to enable compression, still for the same 10K records it has occupied 10MB.

“topic.creation.default.compression.type” : “snappy”

kindly advise how to make this work inside source connector.

6

It’s more important to first ascertain if the compression setting has actually been applied, than compare the size of records on disk.

Can you show the output of kafka-topics --bootstrap-server your_broker:9092 --topic your_topic --describe please?

And also run the test again and find in the Kafka Connect worker log the line INFO Created topic and paste that line here too.

I’m basing this on the blog that I wrote and the details that Kafka shows at the time of creating a new topic.

1 Like
7

Yes i have checked this. Compression is not applied while describing the Topic even when i used the above “topic.creation.default.compression.type” : “snappy” in the source connector.

8

Did you try this? Can you post the results?

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.