High "purgatorysize" in "delayedoperationpurgatory" metrics

We monitor “Delayed Operation Purgatory” metrics of our brokers, and we found the the “purgatorysize” number can be very high during busy time, it can be over 250K.

I have tried a few changes on client applications, e.g. adjust “fetch.max.wait.ms”, but it does not help on this issue, anyone has idea what other metrics that I should look into, or any configs that I should adjust?