I am using the HTTP Source Connector to pull data from an external REST API. The data is returned and using the Expand JSON SMT to build the schema. The problem is that the external source has fieldnames that do not conform with what is allowed (e.g. have a . in the fieldname or a space. Fieldname…

[image] rvaneynd: Struct{values=Struct{PM2.5=9,Temperature=24.9,Relative Humidity=23.9,Air Pressure=1025.1,TVOC=682,CO2=515,CO=0,Ozone=8.3,NO2=43.7,serialNumber=,Virus Index=6,Timestamp=1618579396,DateTime=2021-04-16 14:23}} Hi @rvaneynd ! Have you tried to do what you need in two steps? I thi…

Hi, @rvaneynd ! If I understood you correctly, you are looking for this SMT: ReplaceField | Confluent Documentation

Whatsupbros, I have tried that. But the problem is that I need to replace it before it is passed to Expand JSON SMT. So at that point is it not yet recognised as a Field (as it is still a simple string). As a result the ReplaceField SMT does not work (e.g. does nothing). Raoul.

Can you share a sample of the source data, which you have before all transformations (before expandjsonsmt), what you have after applying them, and the desired resulting data, which you would like to see? It would probably easier to assist you then…

whatsupbros, Thank you. I will do this tomorrow or Saturday. Raoul

Can you also share the connector config that you’re using too please

Whatsupbros, Robin, Thank you for trying to help. Okay this is what I get back from the HTTP Source Connector: {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"HTTP Record Value","field":"value"},{"type":"string","optional":true,"doc":"HTTP Record Key","field":"key"},…

Heh, this is a fun one. You’re hitting up against the issue that JSON has a schema when you eyeball it, but the converters in Kafka Connect deal in explicitly declared schemas. So the HTTP Source connector returns a single field (which happens to hold a JSON object). You can’t process that with any…

Robin, Thank you for the quick response. I thought this was the case… :slight_smile: So what I have done is just to consume it as a plain string in to kafka and kafka has a single string schema. We use azure DataBrick upstream. Using confluents example as the bases for our upstream Azure Databri…

Thank you. It looks like what I am after and will give it a try later. Once again, thank you for trying to help. I will report back ASAP. Raoul.

@whatsupbros , Well some very interesting results. I set it up this way: transforms=HoistValue,Expand,Flatten,RenameFields transforms.HoistValue.type=org.apache.kafka.connect.transforms.HoistField$Value transforms.HoistValue.field=values transforms.Expand.type=com.redhat.insights.expandjsonsmt.E…

Is there an SMT to change field names

Kafka Connect

system Closed 26 April 2021 18:21 16

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ReplaceField SMT does not work after flatten SMT Kafka Connect	1	52	27 November 2024
Csv file with illegal character for avro / rename fields SMT Kafka Connect	6	4588	8 July 2021
New blog: 🎄 Twelve Days of SMT 🎄 - Day 10: ReplaceField Kafka Connect	1	3246	9 February 2021
How to convert nested Json Array to kafka connect struct and schema through SMT Kafka Connect	2	4179	18 December 2021
JDBC source connector support for JSON nested structure Self-Managed Connectors	4	4231	17 April 2024

Is there an SMT to change field names

Related topics