HTTP Sink Connector HTTP Endpoint Issue

Kafka Connect
1

Hi Everyone,

I am new to Kafka Connect. we have decided to use HTTP Sink connector for one of our application and doing a POC around it. We are facing few issues in HTTP Kafka sink connector at the Rest End Point which is created in .net.

HTTP Sink connector is configured successfully and throwing the data to Rest end point. Now at the rest end point we are having issues in receiving the message.

The data in Kafka topic is published as per Avro schema so we are using Avro converter in the sink connector. At the rest end point we have created a dto from the avro schema using avrogen. When the http sink connector throws data to the end point it’s giving “415 Unsupported media type” error.

We wanted to know how the .net api will receive the data, what request object we need to use to capture the data sent by http sink connector. Would request you to please help us with the same or if there is any sample basic project please let provide us the details.

Avro Schema:

{
	"fields": [
		{
			"name": "id",
			"type": "string"
		},
		{
			"name": "name",
			"type": "string"
		}
	],
	"name": "article",
	"namespace": "test.data",
	"type": "record"
}

HTTP Sink connector config:

{
"name": "HTTPSinkConnector_1",
"config": {
    "topics": "test_data_topic_1",
    "tasks.max": "1",
    "connector.class": "io.confluent.connect.http.HttpSinkConnector",
    "http.api.url": "http://localhost/messageHandler",
    "request.method": "post",
    "key.converter":"org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "http://schema-registry:8081",
    "confluent.topic.bootstrap.servers": "kafka:19092",
    "confluent.topic.replication.factor": "1",
    "reporter.bootstrap.servers": "kafka:19092",
    "reporter.result.topic.name": "success-responses",
    "reporter.result.topic.replication.factor": "1",
    "reporter.error.topic.name": "error-responses",
    "reporter.error.topic.replication.factor": "1"
}
2

The connector consumes records from Kafka topic(s) and converts each record value to a String or a JSON with request.body.format=json

You seem to be missing that property.

1 Like
3

Thanks a lot. This is working fine.

1 Like
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.