Issues with optional fields when using Avro schema

I’m trying to create a schema with optional fields. Here’s my setup:

$> cat mytest.avsc
{
  "type": "record",
  "namespace": "com.example",
  "name": "MyTest",
  "fields" : [
    {"name": "field1", "type": "string"},
    {"name": "field2", "type": ["null", "string"], "default": null}
  ]
}
$> cat sample1
{"field1": "value1"}

$> cat sample2
{"field1": "value1", "field2": "value2"}

I’ve also created the topic with the same schema. When I try to ingest sample1 or sample2, I get the following errors.

$> kafka-avro-console-producer --topic mytest --bootstrap-server localhost:9092 --property value.schema="$(< mytest.avsc)" < sample1
org.apache.kafka.common.errors.SerializationException: Error deserializing json {"field1": "value1"} to Avro of schema {"type":"record","name":"MyTest","namespace":"com.example","fields":[{"name":"field1","type":"string"},{"name":"field2","type":["null","string"],"default":null}]}
Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got END_OBJECT
        at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:514)
        at org.apache.avro.io.JsonDecoder.readIndex(JsonDecoder.java:433)
        at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:283)
        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
        at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
        at io.confluent.kafka.schemaregistry.avro.AvroSchemaUtils.toObject(AvroSchemaUtils.java:142)
        at io.confluent.kafka.formatter.AvroMessageReader.readFrom(AvroMessageReader.java:121)
        at io.confluent.kafka.formatter.SchemaMessageReader.readMessage(SchemaMessageReader.java:316)
        at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:51)
        at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

$> kafka-avro-console-producer --topic mytest --bootstrap-server localhost:9092 --property value.schema="$(< mytest.avsc)" < sample2
org.apache.kafka.common.errors.SerializationException: Error deserializing json {"field1": "value1", "field2": "value2"} to Avro of schema {"type":"record","name":"MyTest","namespace":"com.example","fields":[{"name":"field1","type":"string"},{"name":"field2","type":["null","string"],"default":null}]}
Caused by: org.apache.avro.AvroTypeException: Expected start-union. Got VALUE_STRING
        at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:514)
...

Is it possible to have some fields optional with Avro schema? I’ve run into some StackOverflow and google group threads with no luck. I’d appreciate any feedback/guidance.

2 Likes

hi try like this

{“field1”: “value1”, “field2”:null}
{“field1”: “value1”, “field2”: {“string”:“value2”}}

Isn’t there a way to make {"field1": "value1"} to work?