How to fetch all failed fields information while validating AVRO schema with payload

I am not able to fetch all the failed field information. The method just returns the first failure attribute in case of AVRO schema, my requirement is to get all failed attributes information after schema validation with payload.

Hi, @prateek2401. First of all, welcome to this community :blush:

I think what you’re trying to do is, use the Apache Avro framework to validate if a record is compatible with a given schema, am I right?

If this is the case, maybe this topic should be moved to Clients instead. There, folks could better assist in the intricacies of the API needed to do what you’re trying to do.

Nonetheless, with Avro, you usually have two ways of serializing and deserializing data. The first one is by statically compiling the schema and generating the required code using the avro-tools compiler. This is the most common option. In this mode, you would need to rely on the exceptions that are propagated until the upper layers. The second option is dynamic. You can programmatically serialize and deserialize data using the DatumReader and DatumWriter components. With these, you may have more flexibility in capturing not only the last but all fields that failed.

@riferrei

1 Like

Hi Thanks a lot for the information. The thing you are talking about DatumReader have you tried it? Because i didn’t get all failed field.
DatumReader . And the below code i have also tried :

JsonAvroConverter converter = new JsonAvroConverter();

// conversion to binary Avro
byte[] avro = converter.convertToAvro(json.getBytes(), schema);

// conversion to GenericData.Record
GenericData.Record record = converter.convertToGenericDataRecord(json.getBytes(), new Schema.Parser().parse(schema));

// conversion from binary Avro to JSON
byte[] binaryJson = converter.convertToJson(avro, schema);

// exception handling
String invalidJson = "{ \"username\": 8 }";    

try {
    converter.convertToAvro(invalidJson.getBytes(), schema);    
} catch (AvroConversionException ex) {
    System.err.println("Caught exception: " + ex.getMessage());
}

But it just returns me one attribute information in exception. Refer to:
GitHub - allegro/json-avro-converter: JSON to Avro conversion tool designed to make migration to Avro easier.

Do you have any working code where you have captured all field information .If you have do paste it here.
Thanks alot for the help :slight_smile:

DatumReader works fine for regular Avro data, not that serialized intitially by the Confluent-provided serializers for the registry.

Can you please help me with the code? As i tried getting all the failed fields but not able to get that. Do you have any working piece of code for getting all the failure while AVRO schema validation against payload ( Payload and schema are generic so we can’t rely on POJO classes as schema is passed at runtime only )

Even without the schema registry, I don’t think Avro gives you every field that violates your schema. You should consider opening a JIRA with the Avro project for such a feature.

Thanks alot. I have raised a ticket in Jira with Avro project.

One more query i have. I checked when we have mutiple attribute failures it is not giving back the first failed field. It gives random one attribute that failed in case of mutiple attributes failure. Is it so?

I highly doubt it’s “random” ; the Avro specification states that fields have order encoding

HI Do you know is there any library or so which converts AVRO schema to JSON Schema?

@prateek2401 Please create a new post for new questions so it can be more easily searched