Hi, I m exploring strategies around setting up confluent RESTproxy. Right now we have restproxy nodes for each Kafka Cluster. There are few times we have seen rest proxy nodes crashing due to large messages i.e…few applications sending large messages than topics actually allow (>max.message.bytes) , the error message is below. Every time this happens rest proxy nodes crash, and all the other applications hosted on the same restproxy cluster are impacted.
Got error produce response in correlation id 12871518 on topic-partition topic-iotstabilize-0, splitting and retrying (0 attempts left). Error: MESSAGE_TOO_LARGE
With this happening more often, there are several conversations around setting up dedicated rest proxy cluster for each application despite they all share the same Kafka Cluster.
Just curious how everyone else dealing with this type of scenarios? Is that normal to have several restproxy clusters for each Kafka Cluster? or if there is a way we can prevent the restproxy clusters going down when we come across something like mentioned above