Event sourcing with kafka

What is the best practice to structure a message for a topic containing different types that need to be sorted.

Example

Topic: user-events Event types: UserCreatedEvent, UserUpdatedEvent, UserDeletedEvent. Those events need to be saved in the same topic and partition to guarantee the order.

Possible solutions I see

  1. Single schema containing all event type fields
  2. Schema containing all event types schemas. {eventId, timestamp, userCreated: {}, userUpdated: {}, userDeleted: {}}
  3. Different schema for event using Avro union

Pro

  1. Easy to implement and process as a stream
  2. Easy to implement, process as a stream and setup required fields for each event type
  3. Every message is an event

Cons

  1. Possible to have many empty fields and it’s not possible to specify required fields per event type
  2. Not clear the message type without inspecting the payload
  3. Difficult to deserialize (GenericRecord)

Are there other possible solutions, how do you normally handle a topic with different message types? How do you process this king of topics?

Any reference to code example is welcome.

Thanks

Don’t have any specific recommendations, but these two blog posts might give you some ideas:

Apologies if you’ve already read them. :slight_smile:

Hi Dave,
Thanks for the reply.

Yes I already read those articles and I have to say that they give you only a partial answer.

The first tells you when is a good idea to save different types into the same topic, and event sourcing is a good fit.
The second, it’s more technical and illustrate the possibility of doing this with Avro union.

But none of them explain in details how to do it with a real example.

I have seen projects on github where they simplified the scenario by creating a single schema, more as a state than actual event (point 1.).
Talking with someone with some experience using kafka, came up with the solution explained at point 2 by nesting the events into a “carrying event”.

I managed yesterday (I will share the solution asap) to use avro union and deserialize the events as GenericRecord and do transformation based on the event type.
Since I didn’t see any solution similar I was curious if I was missing something, like drawbacks (e.g. Ksqldb doesn’t support different types) or better practices to do the same in kafka.

I used several Avro messages before, and that worked quite nice, as in GitHub - gklijs/bob2021: Repo used for a tutorial during BOB 2021, https://bobkonf.de/2021/klijs.html. I also went full-on using Kafka there, which has some downsides. Not sure If you really need Avro? With protobuf you can easily define multiple message types in one proto file, and use those for your topic. This means you don’t need to use unions, and under the hood the different types are efficiently encoded. It had the major advantage of easily reuse messages as part of other messages. It’s also possible with Avro and references and/or unions, but I feel like that’s a lot more difficult.