I tried searching for Kafka Connect source connectors on confluent hub but wasn’t able to find what I’m looking for.
I want to have a Kafka Connect source connector that listens to http posts from my “legacy” application. The legacy application posts xml content (soap requests) which I want to push to a Kafka topic and then consume from there.
Kai Wähners blog post discusses the possibility of using Kafka Connect for web services (including SOAP) but doesn’t go into details.
The confluent hub link above finds nothing when looking for soap. When looking for http I find this: HTTP Source Connector | Confluent Hub
But it looks like something to poll an http API and not helpfull for my use case where I need to listen to incoming http posts.
So is the only way forwards to start looking into building a Kafka connect source connector or did I miss something that I could use to jump start the process?
Kai Wähners blog post discusses the possibility of using Kafka Connect for web services (including SOAP) but doesn’t go into details.
That’s correct. Actually, I don’t go into details because there is no good web service integration. The only way is to use the XML parsing options I described. This is NOT real web service integration, but more like using the XML message from the web service.
Hence, I guess the two best options:
Either use your existing middleware / SOAP tooling as long as you need these SOAP web services, or re-write the service to use REST or any other more Kafka-native protocol. Or as Dave recommended, put another “transformation service” in the middle (but this could also be the Kafka Connect XML connector.
Probably not what you like to hear, but SOAP-Kafka-integration is not fun.