# Event sourcing with kafka **URL:** \ **Category:** Kafka Streams\ **Created:** [7 April 2021 09:38 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413 "2021-04-07T09:38:58Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [7 April 2021 09:38 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/1 "2021-04-07T09:38:58Z")
What is the best practice to structure a message for a topic containing different types that need to be sorted. **Example** Topic: user-events Event types: UserCreatedEvent, UserUpdatedEvent, UserDeletedEvent. Those events need to be saved in the same topic and partition to guarantee the order. **Possible solutions I see** 1. Single schema containing all event type fields 2. Schema containing all event types schemas. `{eventId, timestamp, userCreated: {}, userUpdated: {}, userDeleted: {}}` 3. Different schema for event using `Avro union` **Pro** 1. Easy to implement and process as a stream 2. Easy to implement, process as a stream and setup required fields for each event type 3. Every message is an event **Cons** 1. Possible to have many empty fields and it’s not possible to specify required fields per event type 2. Not clear the message type without inspecting the payload 3. Difficult to deserialize (GenericRecord) Are there other possible solutions, how do you normally handle a topic with different message types? How do you process this king of topics? Any reference to code example is welcome. Thanks ---
**Author:** ![daveklein](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/daveklein/32/140_2.png) [@daveklein](https://forum.confluent.io/u/daveklein)\ **Post date:** [7 April 2021 21:04 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/2 "2021-04-07T21:04:15Z")
Don’t have any specific recommendations, but these two blog posts might give you some ideas: > **[Should You Put Several Event Types in the Same Kafka Topic? | Confluent](https://www.confluent.io/blog/put-several-event-types-kafka-topic/)** > > Should you to use one one big Kafka topic, or several small ones? Martin Kleppmann lays out a set of guidelines for addressing this problem. > **[Multiple Event Types in the Same Kafka Topic - Revisited](https://www.confluent.io/blog/multiple-event-types-in-the-same-kafka-topic/)** > > Learn how to put several event types in the same Kafka topic using schema references, along with pros and cons. Apologies if you’ve already read them. 🙂 ---
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [8 April 2021 07:16 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/3 "2021-04-08T07:16:48Z")
Hi Dave, Thanks for the reply. Yes I already read those articles and I have to say that they give you only a partial answer. The first tells you when is a good idea to save different types into the same topic, and event sourcing is a good fit. The second, it’s more technical and illustrate the possibility of doing this with Avro union. But none of them explain in details how to do it with a real example. I have seen projects on github where they simplified the scenario by creating a single schema, more as a state than actual event (point 1.). Talking with someone with some experience using kafka, came up with the solution explained at point 2 by nesting the events into a “carrying event”. I managed yesterday (I will share the solution asap) to use avro union and deserialize the events as GenericRecord and do transformation based on the event type. Since I didn’t see any solution similar I was curious if I was missing something, like drawbacks (e.g. Ksqldb doesn’t support different types) or better practices to do the same in kafka. ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [8 April 2021 08:40 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/4 "2021-04-08T08:40:58Z")
I used several Avro messages before, and that worked quite nice, as in [GitHub - gklijs/bob2021: Repo used for a tutorial during BOB 2021, https://bobkonf.de/2021/klijs.html](https://github.com/gklijs/bob2021). I also went full-on using Kafka there, which has some downsides. Not sure If you really need Avro? With protobuf you can easily define multiple message types in one proto file, and use those for your topic. This means you don’t need to use unions, and under the hood the different types are efficiently encoded. It had the major advantage of easily reuse messages as part of other messages. It’s also possible with Avro and references and/or unions, but I feel like that’s a lot more difficult. ---
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [3 May 2021 08:06 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/5 "2021-05-03T08:06:41Z")
This is how I solved deserialization of different Avro schemas in the same topic. Still have problem with type Date. > ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [3 May 2021 10:03 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/6 "2021-05-03T10:03:28Z")
Any reason to not use specific classes and just use `instanceof` to get the class? ---
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [3 May 2021 11:08 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/7 "2021-05-03T11:08:14Z")
Can you do an example? ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [3 May 2021 11:18 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/8 "2021-05-03T11:18:24Z")
This is an example in Kotlin with Spring Cloud [obm\_confluent\_blog/MoneyTransferProcessor.kt at 691d0a70b60b79829be9e2b9ea0a41dda65d35eb · gklijs/obm\_confluent\_blog · GitHub](https://github.com/gklijs/obm_confluent_blog/blob/691d0a70b60b79829be9e2b9ea0a41dda65d35eb/command-handler/src/main/kotlin/nl/openweb/commandhandler/MoneyTransferProcessor.kt#L27). It’s using a specific serde, [obm\_confluent\_blog/application.yml at 691d0a70b60b79829be9e2b9ea0a41dda65d35eb · gklijs/obm\_confluent\_blog · GitHub](https://github.com/gklijs/obm_confluent_blog/blob/691d0a70b60b79829be9e2b9ea0a41dda65d35eb/command-handler/src/main/resources/application.yml#L11). There are different ways to do this. Also important is to have the compiles classes in the classpath. In the example this is done via [obm\_confluent\_blog/avro\_compile.clj at 691d0a70b60b79829be9e2b9ea0a41dda65d35eb · gklijs/obm\_confluent\_blog · GitHub](https://github.com/gklijs/obm_confluent_blog/blob/691d0a70b60b79829be9e2b9ea0a41dda65d35eb/topology/env/dev/clj/nl/openweb/dev/avro_compile.clj) and including the result as dependency. ---
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [3 May 2021 11:55 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/9 "2021-05-03T11:55:25Z")
Understood what you mean. I did try that way first but I was getting AvroTypeExceptions. Example: `if (eventLocation instanceof EventPatientCheckedIn) ` ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [4 May 2021 07:26 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/10 "2021-05-04T07:26:23Z")
Yes so you should use the correct serde and/or set the correct setting. And supply the compiled classes, but makes it a lot easier. ---
**Author:** ![gpietro](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gpietro/32/685_2.png) [@gpietro](https://forum.confluent.io/u/gpietro)\ **Post date:** [4 May 2021 07:33 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/11 "2021-05-04T07:33:59Z")
I’m using GenericAvroDeserializer, what do you mean by using the correct serde? It’s there a way to know the specific Serde for each topic message that I’m reading? Your example seems a bit different because you process different topic but each topic has only one schema type and a specificSerde or maybe am I missing something? ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [4 May 2021 07:51 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/12 "2021-05-04T07:51:44Z")
There’s a bunch of them I gues you can set ` (.put KafkaAvroDeserializerConfig/SPECIFIC_AVRO_READER_CONFIG "true")` like I did in the KafkaAvroDeserializer. ---
**Author:** ![gklijs](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.confluent.io/gklijs/32/223_2.png) [@gklijs](https://forum.confluent.io/u/gklijs)\ **Post date:** [4 May 2021 08:18 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/13 "2021-05-04T08:18:32Z")
As you can see in [schema-registry/avro-serde/src/main/java/io/confluent/kafka/streams/serdes/avro at master · confluentinc/schema-registry · GitHub](https://github.com/confluentinc/schema-registry/tree/master/avro-serde/src/main/java/io/confluent/kafka/streams/serdes/avro) there is a bunch of them. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex019/uploads/confluentcommunity/original/1X/c49438c90c9df282e9996fdf6971be890c71b65a.svg) [@system](https://forum.confluent.io/u/system)\ **Post date:** [19 May 2021 06:50 UTC](https://forum.confluent.io/t/event-sourcing-with-kafka/1413/14 "2021-05-19T06:50:46Z")
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.