I am currently discovering Kafka capabilities and more than a technical question I have a conceptual question so thought this “Architecture and Design” category was a good fit for it.
I have been designing asynchronous based integrations for the last 15 years and depending on the content of the payload being transmitted (just a way to categorize them) they could be:
Message based: The payload contains the whole data set of the entity being transmitted (e.g. all the details of a customer).
Event based: The payloads contains just information about the an event (notification) occurred on a data entity but without the details all the details of the data entity (e.g. customer 123 created or customer 123 updated). This is normally is accompanied by a request-reply read API/Service that provides subscribers the ability of retrieve the data they want upon specific business events.
Two questions:
Reading about Kafka and watching some of the tutorials or webinars (for example yesterday I watched “Event-Driven Architectures Done Right”), I have observed that Kafka talks about events but not in the same terms as I am used (refer to point 2). They talk about scenarios where the whole data sent is being transmitted and anyway they are referring to this as an event driven architecture. Could someone kindly clarify?
In your experience designing asynchronous data exchanges with Kafka, how would you decide about going for option 1 above (message based approach) or option 2 above (event based approach + Read API/Service) for enabling a data exchange between publishers and consumers?
I believe that most people working with Kafka would define an event as consisting of both notification and state, which seems to be your definition of a message. So, in designing asynchronous data exchanges with Kafka I would use option 1, but I would refer to it as “event-based”.
In my experience Option 1 is referred to as “Event-Carried State Transfer”.
As is common in the software engineering field, I think these design decisions are are case-dependent. Some resources that I find useful in these decisions include Martin Fowler’s blog, and here is his post on various Event Driven designs: What do you mean by “Event-Driven”?
Regarding your second question, in my experience it often ends up being both.
It really depends on what needs to happen. For example does the service reading the message need the actual state now, or is the state included in the message enough.
Who you do and don’t put in a message is often a tradeoff. Especially since you don’t really know future use. Keeping them small also helps with performance. Something I did with a client using Mongo and CDC, was to try to extract only the information useful for other services and put that in a schema.
