Confluent cloud REST proxy for producing data

Clients REST Proxy
1

Hello,

I am using confluent cloud and can see REST Endpoint in cluster settings. I believe it just supports v3 services of confluent’s REST proxy. I’m referring to CC API reference but I do not see anything related to producing record to topic with REST proxy.

My intention is to check if via REST endpoint can clients produce data with validating schema registry both AVRO as well as JSON data?

In every document of data prodution via REST PROXY v1 I can see just AVRO validating schema not JSON data. Even for AVRO I tried these steps but getting following error :

{
“error_code”: 405,
“message”: “HTTP 405 Method Not Allowed”
}

Please help me with the documentation or steps to test these scenarios.

2

Hi @shubhamshirur

afaik it’s not implemented

see Can't produce to Confluent Kafka topic through Kafka REST API - Stack Overflow

maybe ksqldb cli could be a workaround, see the blog post by @rmoff

best,
michael

1 Like
3

Thanks for the quick response.

I thought implementing REST server as a separate component on prem pointing to confluent cloud would help, now I am able to produce data via REST endpoint on prem into confluent cloud referring v2 implementation.

But if I try to invoving AVRO messages or JSONSCHEMA (i.e. with schema registry), it is throwing error

After manually adding schema and trying to add with schema id reference.

{
    "error_code": 40801,
    "message": "Error when fetching schema by id. schemaId = 100175"
}

Before registering schema, trying to produce with schema itself.

{
  "error_code": 40801,
  "message": "Error when registering schema. format = AVRO, subject = dummy-value, schema = {\"type\":\"record\",\"name\":\"User\",\"fields\":[{\"name\":\"name\",\"type\":\"string\"}]}"
}

But if I am not adding AVRO or JSONschema, it is simply getting produced into topic.

Not sure if schema registry issue or cloud specific?

If it is resolvable I can go with it, in parallel I am trying solution provided by you mentioned in @rmoff’s blog,

Can you please guide here.

Thanks.

4

Also, not sure why but I am getting 403 CORS rejected request - invalid origin error when I am trying to POST request as shown in @rmoff’s blog you mentioned in conversation.

Please enlight in this case.

5

did you login properly?

6

hey
maybe you schema is wrong
how does it look like?

best,
michael

8

I could validate and save schema to the topic… and tried to use its schema registry.
It is simple dummy schema.

{
	"name": "User",
	"type": "record",
	"fields": [
	{
		"type":"string",
		"name":"name"
	}
	]
}

I think this is not the problem with schema. Otherwise schema registry would not have allowed me to register.

And yes I think I no problem in authentication too, because when I checked ksql-endpoint/info step it is giving me result. Just issue while operations.
Not sure where I am mistaking.

9

And as I mentioned I am able to produce data via self managed REST PROXY into confluent cloud.

But throwing error while registering schemas, not sure why.

10

ok I see
how does you rest proxy config look like?

how do you try to produce the data?

best,
michael

11

REST Config :

bootstrap.servers=xxxxxxxxxxxx:9092
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username=“” password=“”;
ssl.endpoint.identification.algorithm=https
sasl.mechanism=PLAIN
client.bootstrap.servers=.cloud:9092
client.security.protocol=SASL_SSL
client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required username=“” password=“”;
client.ssl.endpoint.identification.algorithm=https
client.sasl.mechanism=PLAIN
schema.registry.url=xxxxxxxxxxxxxxxx
basic.auth.credentials.source=USER_INFO
schema.registry.basic.auth.user.info=schema-registry-api-key:schema-registry-api-secret

Referring this.

And trying to produce exactly like this.

For JSON - this is working fine.
Thanks.