Avro Alternative Becuase Of No Write Access to Broker

Kafka Connect
1

Message: Simple JSON message (no schema).

I worked through the tutorial " Kafka Connect in Action: JDBC Sink" on YouTube with success; however, in my particular use case, I do not have write permission to the broker in order to create a new topic (i.e. Avro stream). I am only allowed to consume the data as it is (simple JSON message/no schema).

Is there a Connect alternative that allows me to create a schema (as in the tutorial using KSQLDB) and append it to the message WITHOUT creating a new topic on the broker?

Thank you.

2

Hi @stcook, welcome to the community :slight_smile:

So that I understand correctly:

  • You have a stream of plain JSON data on a Kafka topic
  • You want to stream it to a database using the JDBC Sink
  • You don’t have the option to add a schema and reserialise the data because you don’t have permission to write the data back to another topic on the cluster

If that’s correct, then I think you might be a bit stuck. The topic that the Kafka Connect sink consumes requires a schema to be present. So you’re going to either need to get permission to write a new topic to the cluster, or have the source topic written using a serialisation format that includes a schema (Avro / Protobuf / JSON Schema).

3

@rmoff , thank you for such a quick response.

Unfortunately, you’re spot on. This case is one where the Kafka service is out of my purview and I can only “subscribe” to the topic.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.