Can't produce to Confluent Kafka topic through Kafka REST API

Clients REST Proxy
1

I have a trouble producing to Kafka topic through Rest API proxy.

I have a running confluent kafka cluster in which I’d like to create a topic and produce a message to it using REST API.

For that purpose I have followed the documentation and created API key and secret.

I manage to create topic:

curl -X POST -H "Authorization: Basic <BASE64_ENCODED_AUTH_KEY_SECRET>"  \
-H "Content-Type: application/json" \
-d "{\"topic_name\":\"test1\",\"partitions_count\":6,\"configs\":[]}" \
https://pkc-xmzwx.europe-central2.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-1zoqz/topics" | jq 

-------------------
returns:
{
  "kind": "KafkaTopic",
  "metadata": {
    "self": "https://pkc-xmzwx.europe-central2.gcp.confluent.cloud/kafka/v3/clusters/lkc-1zoqz/topics/test1",
    "resource_name": "crn:///kafka=lkc-1zoqz/topic=test1"
  },
  "cluster_id": "lkc-1zoqz",
  "topic_name": "test1",
  "is_internal": false,
  "replication_factor": 0,
  "partitions_count": 0,
  "partitions": {
    "related": "https://pkc-xmzwx.europe-central2.gcp.confluent.cloud/kafka/v3/clusters/lkc-1zoqz/topics/test1/partitions"
  },
  "configs": {
    "related": "https://pkc-xmzwx.europe-central2.gcp.confluent.cloud/kafka/v3/clusters/lkc-1zoqz/topics/test1/configs"
  },
  "partition_reassignments": {
    "related": "https://pkc-xmzwx.europe-central2.gcp.confluent.cloud/kafka/v3/clusters/lkc-1zoqz/topics/test1/partitions/-/reassignment"
  },
  "authorized_operations": []
}

MY PROBLEM: I can’t produce to that topic (can’t produce to ANY topic through Kafka Rest API) :

curl -X POST -H "Authorization: Basic <BASE64_ENCODED_AUTH_KEY_SECRET>"  \
-H "Content-Type: application/json" \
--data "{"records":[{"value":"S2Fma2E="}]}" \
"https://pkc-xmzwx.europe-central2.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-1zoqz/topics/test1"  
    

-----------------
returns:
    {"error_code":405,"message":"HTTP 405 Method Not Allowed"}

ALSO TRIED LIKE THIS:

curl -X POST -H "Authorization: Basic <BASE64_ENCODED_AUTH_KEY_SECRET>"  \
                   -H "Content-Type: application/json" \
                    -d  "{ \"key\": {\"value\": \"S2Fma2E=\"} }" \
                   "https://pkc-xmzwx.europe-central2.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-1zoqz/topics/test1/records"  

----------------
returns the same:
{"error_code":405,"message":"HTTP 405 Method Not Allowed"}

IDK if this has something to do with ACL management? Looking into that right now…

Any help would be highly appreciated! :slight_smile: Thanks!

2

Are you using a a dedicated cluster? I believe this is currently only enabled on Dedicated public and peered clusters; it does not currently work on multi-tenant or PrivateLink clusters. Also note that this is currently early access (not GA) in cloud.

Additionally, while your endpoint is correct (you do need /records at the end of the path, so your second attempt is the correct endpoint), I think the structure of your message is slightly incorrect. For example:

curl -X POST \
  -H "Content-Type: application/json" \
  -H  "Authorization: Basic ABC123ABC” \
  "https://pkc-lzvrd.west4.gcp.confluent.cloud:443/kafka/v3/clusters/lkc-vo0pz/topics/testTopic1/records" \
  -d '{"value": {"type": "BINARY", "data": "SGVsbG8gd29ybGQhIAo="}}'

In your case, I don’t believe “value” should be a child of “key”:

{
  "value": {
    "type": "BINARY",
    "data": "S2Fma2E="
  }
}

{
  "key": {
    "value": "S2Fma2E="
  }
}

If you’re using a dedicated (public internet/peered) cluster, and you’re still experiencing issues, you should open a ticket.

1 Like
3

@jlee , does this work with a self managed Kafka REST Proxy? I created another topic for this