I think I got the solution from excellent discussion thread below on Kafka Ordering via different solutions. But I would like to double check whether my use case analysis is correct or not. People are active on this forum, need some quick help on this, so posting here.
Use case: Database is sending the messages on different status:
- ORDER RECEIVED
- ORDER ASSIGNED
- ORDER PROCESSED
- ORDER COMPLETED
We do not have any ordering defined currently, due to the latency on the database side, we are receiving the ORDER PROCESSED (3rd one) message first than ORDER RECEIVED (1st one) due to which the consumer is consuming the same and sending the same WRONG ordered data to downstreams.
Topic has multiple partitions. So my solution is suggesting below changes on both database side and Kafka side:
Database side: Suggest the database team to append the timestamp field to the message while sending to KAFKA.
Kafka side: Kafka Producer config should be changed to include below settings to support no duplication and enables an idempotent producer which ensures that exactly one copy of each message is written to the brokers, and in order. enable.idempotence=true acks=all
NOTE: No need to have single partition topic, rather in the current topic (with multiple partitions), I think I can achieve the strict ordering with above solution.
Appreciate your comments.