I don’t believe that this connector supports starting from an arbitrary offset. The closest thing I see is a azure.eventhubs.partition.starting.position property that lets you start from the beginning or end of the stream. If you’re going “under the hood” and updating connector state, you could try setting offset to the offset or sequence number (depending on what azure.eventhubs.offset.type is set to). I think just putting the number in that field would work, but to be sure, you could run the connector for a bit so that the connector updates the field to non-null and see what it looks like.
Is the source code for this or the self hosted version of this plugin available anywhere? There are other things that I’d also like to understand about it. Like how it produces the partitioning key. The documentation does not cover that. Ideally I’d like to be able to have a one to one mapping between eventhubs and kafka partitions.
The source code is not publicly available. There is a self hosted version here on Confluent Hub (documentation here). Also I see a way to test it out end to end using kafka-docker-playground.
There is documentation on this here. The connector uses “the Azure SDK EventData.PartitionKey property” (i.e., this). That can be null, in which case:
The Azure Event Hub Source connector uses x-opt-kafka-key key-value pair as the partition key when the Azure SDK EventData.PartitionKey property is null and when the x-opt-kafka-key is present.
If x-opt-kafka-key is missing, I believe it’ll give up and go with null.
Thank you for the reply, unfortunately I don not have access to the event hubs cluster and cannot configure EventData.PartitionKeyor see its value. Therefore I don’t know how the connector produces the kafka partition key.