If I set acks=0 in Kafka, my producer will write messages one after another without waiting for any acknowledgment from the leader broker or the replicas/followers. However, my question is whether a consumer can start reading these messages before the leader synchronizes with the followers.
Specifically, if the producer sends messages continuously, can the consumer read them immediately, or does the leader need to synchronize with the followers first before the consumer can access the messages?
Could you please point me to the relevant Kafka documentation that addresses this?
Yes, in the acks=0 case, consumers can consume messages potentially before they are replicated. Here is the relevant Kafka documentation:
Producers, on the other hand, have the option of either waiting for the message to be committed or not, depending on their preference for tradeoff between latency and durability. This preference is controlled by the acks setting that the producer uses. Note that topics have a setting for the “minimum number” of in-sync replicas that is checked when the producer requests acknowledgment that a message has been written to the full set of in-sync replicas. If a less stringent acknowledgement is requested by the producer, then the message can be committed, and consumed, even if the number of in-sync replicas is lower than the minimum (e.g. it can be as low as just the leader).
The guarantee that Kafka offers is that a committed message will not be lost, as long as there is at least one in sync replica alive, at all times.
Hello, dtroiano! I appreciate your response, and I’ve read that part of the documentation. I understand that messages can be consumed by the consumer before the leader synchronizes with the followers. However, my question was more about the meaning of “potentially can.”
Specifically, as long as acks=0, can the consumer always consume messages without waiting for the leader to synchronize with the replicas, or does this only happen in some cases? In other words, is there any situation where a consumer might have to wait for replication, even when acks=0?
Hi @CiprianAmza my original response here was off.
There is the Kafka concept of high watermark that is the point up to which a consumer can read, e.g., see here. The high watermark will only advance after replication to all members of the ISR (in-sync replicas) completes. So, in the happy path where you have n replicas and they are all in-sync replicas, a message will not be available for consumption until it has been replicated across all n replicas. This is true regardless of the producer’s acks setting.
The first wrinkle around this is that the set of in-sync replicas can shrink as they go out of sync (get behind on replication or brokers go offline). (Note that this is independent of the min.insync.replicas config that protects the write path, i.e., will cause an exception during producing if there aren’t enough in-sync replicas acknowledging the message.) When this happens, say the ISR list shrinks to just the leader, then high watermarks will advance and consumers can consume without replication happening. This is the specific situation that is documented here:
If a less stringent acknowledgement is requested by the producer, then the message can be committed, and consumed, even if the number of in-sync replicas is lower than the minimum (e.g. it can be as low as just the leader).
This is the test that I had run that led me to incorrectly believe that the high watermark would always advance in the acks=0/1 cases. I had killed a broker where a partition follower resided and was able to continue consuming. I was actually in that limited shrunken ISR scenario though.
The second wrinkle that is hotter off the press is that this behavior changed as of Apache Kafka 3.7 / Confluent Platform 7.7. As part of KIP-966 to improve data durability in some specific failure scenarios, KAFKA-15583 changed the behavior such that the high watermark will only advance if the min ISR is met. I.e., whereas before min.insync.replicas strictly provided produce path protection, now it also provides some consume path protection to address the data loss scenario described in the KIP. Given this, setting min.insync.replicas=1 on Kafka >= 3.7 results in the same consume path behavior as in Kafka < 3.7 (though not the same produce behavior / protection).
We are going to update the documentation to clear up this new behavior related to min.insync.replicas.
I appreciate your response, and I’ve read that part of the documentation. I understand that messages can be consumed by the consumer before the leader synchronizes with the followers. However, my question was more about the meaning of “potentially can.”