How to design topics/messages correctly

Stream Processing
1

Hey folks, I have a question about how to “design” Kafka topics/messages.
I am trying to find the correct balance.

Context first:

  • SystemA - producer
    • There are entities (db tables) with relationship between them. For example, EntityA has multiple EntitiyB, and Entity B has a single EntitiyC and so on.
  • SystemB/SystemC/… - consumers

Requirement:

  • I want the SystemA to produce messages on each entity change (create/update/delete).
  • The number of entities is small, and the changes frequently are also small. Let’s say thousands per year (probably much less than that)

I thought about the following:

  • a dedicated topic per entity events, for example: entitiyA.events , entityB.events ,…
  • Each topic with 1 partition
  • The message will be JSON looks like:
{
	"id": UUID
	"type": "entitiyA:created"
	"sent_at": DATETIME
	"body": {
	    "id": 1687,
	    "foo": "bar",
	    ...
	    ...
	    ...
	    ...
	}
}

Questions:

  • What do you think about the above design?
  • I know there is the approach of a single topic for all the messages across all entities- is there a reason to do that over a dedicated topic per entity? It’s all about balance…

Happy to hear your experienced thoughts

2

Hey there! I agree with you about having a dedicated topic per events. Here’s a post I wrote that says more on this “topic” haha. Happy to hear other weigh in on change data capture strategy as well and will keep an eye on this thread.

1 Like