Imagine I have a number of microcontrollers connected to a network. They are each a consumer awaiting events in order to perform actions. They also produce events when they perform an action. However, they’ll only perform actions when they receive events from certain controllers. If controllers A or D produce events after their action then controller F will perform an action, etc. F is inert to any events from any other controller.
My initial thought was to have each controller consume the same topic, with the producer id in the payload, then the code will determine which events to action on. However, if I had hundreds of controllers then all controllers will consume all events, which seems to be wasteful if they only care about a small fraction of them.
The other idea is to have multiple topics, with the source controller id in it. So the target controller will just consume their target controller’s messages. But this seems complicated to setup.
So I’m wondering if there’s a feature in Kafka that will allow producers and consumers to be connected, like a graph, but indirectly through the broker?
Btw, this is somewhat a thought experiment. I don’t think Kafka would be a practical tool in this case. But, if it is, then Kafka is even more awesome than I thought.