# Kafka Connector, both a source and a sink **URL:** \ **Category:** Kafka Connect\ **Created:** [30 January 2025 15:17 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900 "2025-01-30T15:17:19Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![nikola.milutinovic](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nikola.milutinovic](https://forum.confluent.io/u/nikola.milutinovic)\ **Post date:** [30 January 2025 15:17 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/1 "2025-01-30T15:17:19Z")
Hi all. I want to connect our system, based on Kafka, to a REST server. The server is able to create a resource and will respond with resource ID. We need to propagate this ID back to the our system, so the resources on our end can be properly linked. Let’s call this resource a Note. 1. So, our system (user interaction) will create our Note. 2. It will be pushed out to Kafka topic “out-note”. 3. Kafka Connect will pick it up and do a “POST /note” on the REST server. 4. REST server will respond with {“id”: “01234-212314-222111-12346”, …} 5. This message should be pushed back to Kafka, topic “in-note-updates”. The last step is my problem. This sounds like the adapter should be both a source and a sink. Which is **not** the regular interface of Kafka Connect. I could try to establish some sort of a back-channel, like writing to a file and using File Source connector, but that is awkward and error-prone. Another option is to supply Kafka credentials to my adapter, so it can push a message to the “in-note-updates” topic. But that also feels wrong. How do people usually solve synchronization with a request/response based service? ---
**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\ **Post date:** [31 January 2025 09:01 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/2 "2025-01-31T09:01:09Z")
Is there a feature of Kafka Connect in particular that you’re looking to make use of in this? It sounds more like you just need a regular Kafka consumer/producer. ---
**Author:** ![nikola.milutinovic](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nikola.milutinovic](https://forum.confluent.io/u/nikola.milutinovic)\ **Post date:** [3 February 2025 08:45 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/3 "2025-02-03T08:45:54Z")
Hi RMoff. The reason I am looking at Kafka Connect are the non-functional goodies it brings, like being able to write the Task, which is the gist of what needs to be done; letting KC do workload balancing and execution management. Logging and observability also come into the picture. But you are right, this is, in essence, a regular 2-way producer/consumer. Is it then recommended to abandon KC for such use-cases? Doesn’t this use case fall under the “charter of integration”? I would expect that I am not the only one who faced a request/response external endpoint and wanted to use KC to integrate. ---
**Author:** ![rmoff](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/rmoff/32/3_2.png) [@rmoff](https://forum.confluent.io/u/rmoff)\ **Post date:** [3 February 2025 15:58 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/4 "2025-02-03T15:58:11Z")
> Is it then recommended to abandon KC for such use-cases? Possibly, yes. Because of this: > Which is **not** the regular interface of Kafka Connect. If it doesn’t fit naturally, then it sounds like it’s the wrong fit. But—perhaps others will have different opinions, so perhaps they will weigh in here 🙂 ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [3 February 2025 19:23 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/5 "2025-02-03T19:23:47Z")
@nikola.milutinovic technically you can have a connector that operates as source and sink, but you have to pick which one to use. For illustration, [this](https://github.com/kakao/kafka-sink-connector/blob/da676e541d47b0eae6870a6aed1622a981f1a7c2/src/main/java/com/kakao/connector/kafka/KafkaSinkTask.java#L87) is a sink connector that copies from one topic to another. You can imagine having a sink connector that reads from the `out-note` topic and whose `put` method makes API calls and writes back to a different Kafka topic. That being said, this makes me think that the Produce / Consume API might be better: > [@nikola.milutinovic](#): > > How do people usually solve synchronization with a request/response based service? I.e., the fact that you are asking about _synchronization_ across topics makes me think that a tightly coupled request / response using Kafka transactions will give you tighter control over failure scenarios. How bad is it if `POST /note` gets called multiple times for the same note? How bad is it if it doesn’t get called for a note? ---
**Author:** ![nikola.milutinovic](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nikola.milutinovic](https://forum.confluent.io/u/nikola.milutinovic)\ **Post date:** [5 February 2025 16:03 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/6 "2025-02-05T16:03:53Z")
@dtroiano Thank you for an example of a 2-way connector. I was thinking along those lines, myself. As for transactional support, you could be on the more correct track, there. We cannot allow a single note to be “lost”. Multiple calls should not be a big problem. However, missing the link between external note ID and internal note is also not acceptable. So, maybe a custom written client that will have the full freedom (and responsibility) is in order. Thank you for sorting thing s out for me. ---
**Author:** ![dtroiano](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/dtroiano/32/1961_2.png) [@dtroiano](https://forum.confluent.io/u/dtroiano)\ **Post date:** [5 February 2025 21:40 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/7 "2025-02-05T21:40:22Z")
👍 this sounds like a good direction. Regarding transactions, you basically have the `consume-process-produce` pattern described [here](https://docs.confluent.io/cloud/current/client-apps/optimizing/durability.html#exactly-once-semantics-eos): > To use transactional semantics in a `consume-process-produce` pattern and ensure each message is processed exactly once, a client application should set `enable.auto.commit=false` and commit offsets manually using the `sendOffsetsToTransaction()` method in the `KafkaProducer` interface. But, offsets management and transactions might not even be required for you since you say “Multiple calls should not be a big problem.” It’ll come down to whether you want the convenience / ease and lower overhead (e.g., of automatic offset management and no transactions) in exchange for higher risk of dupe calls in failure scenarios. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [7 March 2025 21:41 UTC](https://forum.confluent.io/t/kafka-connector-both-a-source-and-a-sink/36900/8 "2025-03-07T21:41:04Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.