Question on SetPartitionsAssignedHandler

Clients Non-Java Clients
1

This question is about SetPartitionsAssignedHandler semantics for Kafka consumers written using C# client library Confluent.Kafka.dll.

We have client/server applications which produce as well as consume messages. Simplified high level architecture is as below:

image
image761×379 6.13 KB

Here:

  1. Client produces a request to RequestTopic and waits for response from ResponseTopic.
  2. Server consumes requests from RequestTopic, computes a result, and produces the result on to ResponseTopic
  3. Client consumes responses from ResponseTopic

So there is a producer and a consumer on both client and server side.

The consumer on the client side is alone and is consuming messages from the latest offset available from the ResponseTopic. The consumer configuration is as below:

var consumerConfig = new ConsumerConfig
{
BootstrapServers = Configuration.BootstrapServers,
MessageMaxBytes = 10000000,
AutoOffsetReset = AutoOffsetReset.Latest,
GroupId = Guid.NewGuid().ToString(),
EnableAutoCommit = true,
EnableAutoOffsetStore = true,
AllowAutoCreateTopics = true,
}

At startup, the client waits for assignment of ResponseTopic. Only post assignment of ResponseTopic the client will start producing requests to RequestTopic. This is so that the clients do not miss response to any request.
The synchronization between producer and consumer on the client side is done based on SetPartitionsAssignedHandler event being raised for the consumer. So, requests will be produced only after SetPartitionsAssignedHandler event handler is called for the response consumer on the client side.

This pattern generally works but in rare cases, even though request was raised only after SetPartitionsAssignedHandler was called, the responses would not be consumed.
On analysis, this was found to be happening when the time difference between ResponseTopic assignment on client side and the Response being produced on the server side was very small (generally 0.1 second or less).

On further analysis we realized from C# Confluent.Kafka client library code, that Assign call for actual assignment of partition is called only post SetPartitionsAssignedHandler event handler is called hence our pattern was not full proof.
So, if the Reponses were produced before actual assign call then there is every chance that response would be missed.

Based on above:

  1. We understand that, SetPartitionsAssignedHandler event handler being raised does not mean that the partition assignment is completed? Is this correct? If so, this is not obvious from API documentation SetPartitionsAssignedHandler.

  2. Is there any way to understand (any event) which gives indication of successful topic, partition/offset assignment?