hello there,
I’m writing a Source connector for an external system which consumes a message with the following json strucure:
"payload": { value="120", type="integer"}
The ‘payload’ field can also contain any other type such as ‘boolean’, ‘double’, ‘string’ for the field ‘value’ but not simultaneously. I’m trying to model the payload field using connect’s ‘SchemaBuilder’ class with the following definition:
public static final Schema SCHEMA = SchemaBuilder.struct().name(NAME)
.version(1)
.field(S_FIELD, Schema.OPTIONAL_STRING_SCHEMA)
.field(L_FIELD, Schema.OPTIONAL_INT64_SCHEMA)
.field(D_FIELD, Schema.OPTIONAL_FLOAT64_SCHEMA)
.field(B_FIELD, Schema.OPTIONAL_BOOLEAN_SCHEMA)
.build();
In my initial tests with ProtobufConverter I can see the optional fields being filled with default values (if not set), e.g getting something like the following using ‘kafka-protobuf-consumer’:
"payload":{"s":"","l":"1091","d":0.0,"b":false}
whereas when using instead the AvroConverter, the missing fields are set to ‘null’.I know Protobuf provides an ‘oneof’ keyword when defining the schema for cases like this and wondering if this can also be achieved with a kafka connect SchemaBuilder definition . If not what is the recommended approach to model this case taking into consideration the need to support all schema types Avro/Protobuf/JsonSchema
oneof payload {
string s = 5;
int64 l = 6;
double d = 7;
bool b = 8;
}
Appreciate any help with this.