Out of order Messages

Hi, Im getting a lot of “Detected out-of-order KTable” update messages in my logs.
I am using kafka streams 3.1.0 running on kubernetes (in ec2) and aws MSK for brokers.
How can I go about debugging this error? the timestamps are set by the producers and not by the broker. Thanks in Advance

If you read a topic via builder.table() data should be in timestamp order per key. If not, you get the corresponding log messages you observe.

It seems that two different upstream producers write messages for the same key, and thus introduce out-of-order data, because their writes are interleaved?

Cf What's the time? ...and why? - Confluent