Confluent HTTP Sink Connector value converter encoding/decoding behaviour

Kafka Connect
1

Hi all :slight_smile:

I have a HTTP Sink connector set up in a distributed setting trying to send data from kafka to an external source. This is currently just a PoC so I want to test data flow all the way form a provider into kafka → kafka streams → kafka connect → external recipient. I am having an issue where the recipient is reporting wrong encoding of special characters (æ,ø,å). From what I’ve read, some Java versions store strings as unicode characters by default. Changing the charset in my deserializers / serializers and my gradle.build file solved the first part of the problem. However, the problem reappears when streaming data through my connector instance.

I have checked my source and output topics as well as data streams through kafka streams. No encoding issues here.

Here’s what I’ve tried so far for the connector instance:

  • Using the org.apache.kafka.connect.storage.StringConverter. (This gives me the encoding problems)
  • Using “value.converter”: “org.apache.kafka.connect.json.JsonConverter”, “value.converter.schemas.enable”: false, (This gives me a bad request body. JSON is sent with “=” instead of “:” as the key/value separator. (why?))

StringConverter behaviour:

{“records”:[{“items”:[{“field”:“@ACCOUNT”,“value”:91351787},{“field”:“@NUMBER”,“value”:91351787},{“field”:“FNAVN”,“value”:“J?RN K?RE GI?VER”},{“field”:“LNAME”,“value”:“J?RN K?RE GI?VER”},{“field”:“CUSTADDRESS”,“value”:“123 Main St”},{“field”:“POSTNO”,“value”:“12345”},{“field”:“POSTPL”,“value”:“123 Main St”},{“field”:“BIRTH_DATE”,“value”:“2023-12-14 00:00:00”},{“field”:“EXP_DATE”,“value”:“2023-12-14 00:00:00”},{“field”:“INS_DATE”,“value”:“2023-11-16 15:40:00”},{“field”:“OPTIONAL_TEXT1”,“value”:“J?rn K?pperud J?ger har alle de kule tegnene fra norsk i navnet sitt”},{“field”:“OPTIONAL_TEXT2”,“value”:“J?rn K?pperud J?ger har fortsatt alle de kule tegnene fra norsk i navnet sitt”},{“field”:“MAINPHONENUMBER”,“value”:91351787},{“field”:“PROJECTDESC”,“value”:“LEADS”},{“field”:“SEGMENT”,“value”:“LEVERAND?R B”}]}]}

JsonConverter behaviour:

org.apache.kafka.connect.errors.ConnectException: java.io.IOException: HTTP Response code: 400, Bad Request, Submitted payload: {records=[{items=[{field=@ACCOUNT, value=99641867}, {field=@NUMBER, value=99641867}, {field=FNAVN, value=J?RN K?RE GI?VER}, {field=LNAME, value=J?RN K?RE GI?VER}, {field=CUSTADDRESS, value=123 Main St}, {field=POSTNO, value=12345}, {field=POSTPL, value=123 Main St}, {field=BIRTH_DATE, value=2023-12-14 00:00:00}, {field=EXP_DATE, value=2023-12-14 00:00:00}, {field=INS_DATE, value=2023-11-16 15:40:00}, {field=OPTIONAL_TEXT1, value=J?rn K?pperud J?ger har alle de kule tegnene fra norsk i navnet sitt}, {field=OPTIONAL_TEXT2, value=J?rn K?pperud J?ger har fortsatt alle de kule tegnene fra norsk i navnet sitt}, {field=MAINPHONENUMBER, value=99641867}, {field=PROJECTDESC, value=LEADS}, {field=SEGMENT, value=LEVERAND?R B}]}]}

2

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.