Hi, I have a question about guaranteed ordering and strategies for handling failure. I know that the order of records is guaranteed within a partition, but are there any products / cloud services / streams libraries, which fulfill the following requirements?
- a failed record should be sent to a dead letter topic and from there can be sent back to the application for the processing to be retried
- any records with the same key must not be allowed to overtake that failed record, rather should be processed in order after it is no longer failing
- any other records on the partition with different keys may not be blocked by the failed record or its related records (the ones with the same key)
Basically, I am looking for partitioning at the individual key level.
I have a prototype based on Kafka streams, but it would be simpler and cheaper in the long run to use something that already exists.
The reason that we need this is that the advantages of serial processing per key outweigh the disadvantages of blocking records with the same key that are created later.