Producer retries

Kafka Streams
1

Hi,

My apology if this is a question which is already been answered before.

Use Case:

I am reviewing an application (producer) which is using librdKafka as a 3rd party library to interact with Kafka brokers (3= 1L+2F). This 3rd party lib has been configured for retries. Its a single topic with multiple partitions. Application suppose to maintain a strict ordering without any loss of messages. What I found is if the library fails to deliver the message after retires (not getting ack from broker) and notifies the application, it then again retries to send.
(idempotent feature is enabled.)

Question:

  1. Will this break the ordering of message (if the application retries)?
  2. Will enabling idempotent going to help here ?
  3. Is the retry from application unnecessary overhead in ALL cases.

Thanx
Abhijit

2

Your question is about librdkafka, and thus it’s best to post in Non-Java Clients - Confluent Community

The “Kafka Streams” category is, well, for Kafka Stream, the Java stream processing library of Kafka: Apache Kafka

3

Its not related to which kafka library application using… its more on the concept of application retry vs producer (library - can be java/python/c++ etc). retry.

4

Not sure if I can follow – from my understanding, it highly depends on how an library concretely implements it’s own retry logic etc… Of course, there is some basic Kafka protocol fundamentals, all client implementations build on top, but in the end, it’s still a lot of client impl dependent semantics.

5

Ok, probably I can simplify my question -
Lets say an application builds messages M1,M2,M3 and uses library to send towards brokers, using a single topic and partition. Its an async call. Assume due to some reason delivery of M3 fails. The library/producer retries, gives up and reports an undelivered error to application. The application in the meantime builds M4 and M5 and uses the library call to send. Then it retries M3 again. Now this time the library was successful to send to broker.
Will it generate M1,M2,M4,M5,M3 towards consumer?

6

Will it generate M1,M2,M4,M5,M3 towards consumer?

Yes, that is reasonable. In the end, if M3 failed and an error is given back the app (and the producer would “drop” M3). If M3 is re-sent by the app, the producer does not even know that send(M3) is a retry of a previous send, and can just “enqueue” it at the end.