Passing data from a topic to source connector

Kafka Connect
1

I am looking to solve the challenge below:

Read value from a topic, call webservice using topic value and write the response to a topic leveraging Kafka connect.

Using the Robin Moffat example as a basis https://www.confluent.io/blog/data-wrangling-apache-kafka-ksql/“rest.source.url”: “http://environment.data.gov.uk/flood-**monitoring/id/stations**/L2404” the url is static.

In my case, I would like to read a value from a topic and place it on the URL stations**/{some_id_from_topic}]**)

My intention is to modify the source connector to read from the topIc.

I am relatively new to Kafka and was interested if there are any alternatives or gotchas to my plan.

2

Hi Garry,

Welcome to the forum!

What you’re describing is possible, but I think that writing a client application would be much simpler. You could use a consumer to read from the topic, use the values to make the web service call, then use a producer to write the results to the target topic. Is there a reason you’d like to avoid running a client application?

Dave

3

Thanks, Dave. My client is looking for a minimal code solution (they are a .net house). I am leveraging KSQLDB and Connect for most of the work. The fact that I could leverage some code from an existing connector and modify it and just pop it into connect would be the ideal solution for me.

The end client is weighing up a custom .net solution via Kafka. Having the connector configurable via rest would keep the solution appealing to them. The whole solution would be made up of connectors and KSQLDB which reduces the number of moving parts.

I am working on a proof of concept where they are more likely to go with Kafka if things are neat and tidy. So while I don’t rule out writing a client app I was hoping to keep things in the Connect/KSQL domain.

Kind Regards,
Garry

4

In that case, what you are suggesting might be a good way to go. At a minimum it sounds like a lot of fun! If you don’t mind, it would be great if you can post back here, and let us know how it worked out.

Thanks,
Dave

5

Hi @Garry ,

We are doing this in our project. Let me know if you need any help.

However, I think it should be a sink rather than a source as it reads from topic and makes a http call.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.