Need a little help regarding Kafka integration in a Power Platform architecture scenario

Certification and Training
1

Hello everyone,

I’m now studying for my GH-300 Certification and came across a practice question involving the integration between Kafka and the Power Platform solution. While I know how the publish-and-subscribe mechanism works in Kafka, this particular question seems to confuse me, which is why I decided to post my question here.

Here is the scenario:

Kafka is utilized by the company for streaming real-time order events coming from an external system. Such events have to be consumed and recorded in Dataverse, allowing business people using Power Platform applications to view real-time updates. In the course of preparing for GH-300 certification, it is necessary for the solution architect to determine how to consume the Kafka events and add them to Dataverse in such a way that it will not be overloaded or duplicates will not occur.

The options I came across are:

A) Have a single consumer read directly from all Kafka partitions and write every event straight into Dataverse using individual API calls as soon as each message arrives.

B) Use a consumer group with multiple consumers reading from Kafka partitions, process events in batches, and write to Dataverse using upsert operations based on a unique event identifier to avoid duplicates.

C) Skip Kafka partitioning entirely and just have every consumer read every partition so that no messages are ever missed.

D) Write Kafka events directly into Dataverse without any consumer group, since Dataverse can handle deduplication on its own automatically.

At first, I felt that the primary problem in this scenario would be just that of consuming Kafka quickly enough, but I’m still unsure as to how the partition assignment within a consumer group can be done in such a manner as to prevent duplicates on the Dataverse end. Considering that this type of integration logic keeps popping up during GH-300 Certification preparation, I felt it was important that I understood the logic behind it rather than merely memorizing it.

Is this properly solved through upserts with the unique identifier, or is there a better way to approach this when using Kafka streams within a Power Platform application?

It would greatly benefit my understanding if I could receive an explanation of what the proper response would be, along with the rationale behind the behavior of the Kafka consumer groups in such a situation.

Thanks in advance for any explanation !!!

2

The correct answer is Option B.

​Here is the crisp breakdown of why this is the only viable architectural approach:

  • ​Consumer Groups (Scalability): Kafka automatically load-balances partitions across the consumers in a group. This allows parallel processing without different consumers reading the same data.
  • ​Batching (API Protection): Writing data in batches prevents the integration from instantly triggering Dataverse’s strict API throttling limits (service protection).
  • ​Upserts (Deduplication): Kafka guarantees at-least-once delivery, meaning network blips can cause a message to be read twice. Using an Upsert (Update or Insert) tied to a unique event ID makes the operation idempotent—a duplicate read simply updates the existing row instead of creating a duplicate.

​Why the other options fail:

  • ​A: A single consumer bottlenecks throughput, and individual API calls will immediately throttle Dataverse.
  • ​C: Multiple consumers reading everything destroys scalability and guarantees massive data duplication.
  • ​D: Dataverse’s built-in duplicate detection is meant for manual UI warnings (like preventing two contacts with the same email), not for high-throughput streaming integrations.