Redshift connector infer schema support

Kafka Connect Self-Managed Connectors
1

Hello,
I started a POC with Redshift connector so as to do incremental load from MSK to Redshift.

Setup was easy and quick. So great job on that!

The thing which I am struggling now is with the schema deducing which is not supported by the connector and hence our data in Kafka stream which is in JSON(only payload) format was not deduced by Connector.

I also came across this article and wondering where are we with this.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-301%3A+Schema+Inferencing+for+JsonConverter

  1. Is Confluent team planning to support this feature and if yes when can we expect it?
  2. Does Json convertor supports deducing records using schema registry? If yes, do we have any documentation mentioning the steps for the same?

I am a beginner, so in case this has been already answered somewhere please let me know the link to follow.

Thanks

2

@sagar KIPs are proposals from the entire Apache Kafka community for ammending the Apache Kafka project itself. I just want to highlight that it’s not entirely up to Confluent to implement various KIPs into the project.

@rmoff went into some details here if some clarity on JSON and JSON schemas might help: Kafka Connect and Schemas

1 Like
3

thanks for the inputs :slight_smile:

One more question, does Redshift connector supports Upserts? And if it doesn’t support then what could be the workaround?

I was able to test deletes, but upserts are not working.

Note: I am using “pk.mode”:“record_key”

1 Like
4

Hello! I have the same question about the upserts. Did you find any solution?