Sharing here this repository that I created on GitHub that contains a complete example of how to create a custom connector for Kafka Connect. This example contains:
Complete source-code of a sample source connector with dynamic re-partitioning.
Docker Compose using Kafka Zookeepeerless from Confluent for local deployment.
Kafka Connect configured to allow remote debugging connection using JDWP.
Terraform code that shows how to programmatically deploy the connector at AWS.
Cloud deployment using Amazon MSK and MSK Connect. Complete example.
Also, very soon I’ll be creating and publishing a complete tutorial about this code on the BuildOnAWS YouTube channel. I’ll discuss the design decisions, schema management choices, the trade-offs, the implementation gotchas, and everything you need to understand about the Kafka Connect SDK. Make sure to subscribe to the channel do you don’t miss it.