New blog: Defining custom topic properties with Kafka Connect source connectors

When Kafka Connect ingests data from a source system into Kafka it writes it to a topic. Assuming you have set auto.create.topics.enable = true then it will create these topics for you, and do so using the defaults configured on the broker for topic creation, including num.partitions and default.replication.factor .

There are many other topic-level configurations which you may want to set for topics that are automatically created by Kafka Connect. This is particularly true for connectors which are creating a large number of topics, or where the topic name is not known in advance (e.g. when using a regex to select objects from the source system) and thus cannot be pre-created with the desired settings. Common settings for a topic that you may want to customise include cleanup.policy , min.insync.replicas , and compression.type .

KIP-158 was implemented in Apache Kafka 2.6 (available with Confluent Platform 6.0), and adds the ability to set topic-level configurations for topics created by Kafka Connect source connectors.


Read more: Defining custom topic properties with Kafka Connect source connectors