Hello,
When creating a new topic with a high number of partitions (e.g., 64) and a replication factor of 3, there is a race condition between the AdminClient confirming topic creation and all partition leaders becoming fully stable/ready for I/O.
We are using Kafka(4.1.1) + KRaft.
If a shared KafkaProducer (with enable.idempotence=true) attempts to publish a burst of messages immediately after AdminClient.createTopics().all().get() returns, it frequently encounters a fatal OutOfOrderSequenceException.
To make it a bit more clear, let’s say I have a KafkaProducer and I start producing messages for topic1. Then I create the topic2, I wait for the adminClient.createTopics(List.of(topic2)).all().get() to succeed and then I start to also produce on topic2. Now, one of two things may happen:
- It works normally
- org.apache.kafka.common.errors.OutOfOrderSequenceException: The broker received an out of order sequence number. Partition: x Offset: -1
Now the second case has 2 possible subcases:
- all partitions are broken and writing is not possible anymore on that topic.
- some partitions are broken, some are not => messages on some partitions are still being written
Our use case is this: on one application we create the kafkaTopic:
adminClient.createTopics(Collections.singleton(newTopic)).all().get();
and on another application, once we receive the confirmation, we start producing messages on that topic instantly. We would expect that "createTopics().all().get()” to only finish when writing is safe on that topic, but it seems that this is not the case.
To solve this, we found various ways:
- set ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to “1”, but this is not desired, as it will slow down the throughput too much
- add a delay/sleep after the adminClient.createTopics().all().get(), but again, here is not clear what timeout would be enough (in some cases 2 seconds was not enough)
- send the first message for each new topic sync (sharedProducer.send(new ProducerRecord<>(topicName, “firstMessage”, (“dummy”).getBytes())).get()
(it seems hackish)
We also take into account to use a new KafkaProducer for each new topic, but I would really like to understand what causes this error and why is adminClient.createTopics().all().get() finishing before the topic is safe to write on?
We verified the topic state using describeTopics before producing. All 64 partitions showed valid leaders and full ISR (In-Sync Replicas). Despite the metadata being stable, the OutOfOrderSequenceException still occurs. This suggests that the issue is not just metadata propagation, but a race condition in the broker’s request processing layer when handling the very first idempotent batches on a cold log file, where a subsequent batch (Seq > 0) is processed before the initial batch (Seq 0) due to initial I/O overhead.
It never happens for the first topic that the KafkaProducer connects to . Is this a known problem? Could anyone please explain me the mechanism behind it? Thanks!
Later Edit for Correction: This issue occurs even for the very first topic produced by a fresh KafkaProducer. This confirms that the problem is not related to state leakage from previous topics, but a fundamental race condition during the initial handshake between an idempotent producer and a newly created partition. Even with a brand new Producer ID, the broker’s inability to handle concurrent initial batches (Seq 0, 1, 2…) results in a fatal exception instead of a retriable recovery.
In order to prevent it, the KafkaProducer has to be instantiated AFTER the topic has been created.