SourceConnector waiting for async events

Kafka Connect
1

Greetings, my first post!

My data-flow model is async events occuring in my external system that I want to turn into Kafka messages to send along to a broker. Example: “2022-07-13T12:13:14454Z|1DKGEVKS0002|SOLD” So I’ve determined that SourceConnector is the pattern to use, of which there are many examples but none use async events as input. I think the logic to accomplish this goes in SourceTask .In a non-Kafka world my little server that waits for these would have a blocking accept method.

Any ideas , or better yet working examples, of this pattern? Or am I using the wrong pattern?

TIA,

Chris

2

I solved this by baking a KafkaProducer into a std ServerSocket that runs as a service - async events come in on client socket connections to the server, which receives them, transforms them into ProducerRecords and sends them on their way.

Phase II would be to create the ProducerRecords in the 3rd-party software using the Kafka wire protocol and send them directly to our cluster - no more service needed.

CASE CLOSED

Chris

3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.