I have a use case that I believe is an excellent fit for Kafka Connect. The use case is as follows:
An external partner delivers events/messages to our IBM MQ at the perimeter of our environment. Those events/messages will then be picked up from MQ and moved to Confluent Kafka where we will use Kafka Streams API to move the events/messages to the correct topics.
For the step to move events/messages from MQ to Kafka, my thought was that the MQ Source Connector would be the best option. However, I’m being told that I shouldn’t use Kafka Connect, and instead write a producer/consumer pair to move the messages from MQ to Kafka. That just doesn’t make sense to me given what I know about Kafka Connect and it’s purpose. Am I thinking about this the wrong way? I apologize if this seems like I’m asking for opinions, but it’s blowing my mind that our Kafka team is telling teams to create producers/consumers for things like MQ, MongoDB, etc. Isn’t that what Kafka Connect is for?
I can’t get a straight answer from the Kafka team on the why behind their position, so I thought I’d check with the community to see if there are documented issues/limitations/whatever with Kafka Connect (or the MQ Source Connector for that matter) that I am just not aware of, and that would steer one away from actually using it as intended.
Thanks.