KsqlDB leverages the connect API for handling structures. This is quite handy to have the object structure syntax that can be converted to various formats to align with connect.
In the Connect API SchemaBuilder is a Schema (it inherits Schema).
I had created a UDAF that returned a structure with an array, but I had forgotten to do build() on the array schema.
This code compiled and registered with KsqlDB. When using KsqlDB within docker-compose I saw no errors (different reason for that, but in any case added to my confusion).
private static final Schema GEO_POINTS = SchemaBuilder.struct()
.optional()
.field("LOCATIONS", SchemaBuilder.array(GEO_POINT).optional())
.build();
After local installation of KsqlDB and a remote debugger, I uncovered the issue, the “LOCATIONS” element of type array was of type SchemaBuilder, not ConnectSchema and two identical schema constructs (one a ConnectSchema type and one a SchemaBuilder type are not equal)
I added in the missing .build() and now everything is working fine.
private static final Schema GEO_POINTS = SchemaBuilder.struct()
.optional()
.field("LOCATIONS", SchemaBuilder.array(GEO_POINT).optional().build())
.build();
In the error log you will see “This could happen if the value was produced by a user-defined function where the schema has non-optional return types” (provided the error log is working), focus on the cause “Caused by: org.apache.kafka.connect.errors.DataException: Struct schemas do not match.” If I had done that, I would have gotten to solving my issue sooner.