org.apache.kafka.common.errors.RecordTooLargeException

gg24 · 14 September 2021 01:45

Hello i am trying to address the large payload size issue, when I try to push large payload(30MB) using offset explorer i am seeing this message

“org.apache.kafka.common.errors.RecordTooLargeException
The message is 3280843 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration”

Here are the properties i have set in different files

-- server.properties
message.max.bytes=104857600
replica.fetch.max.bytes=104857600

-- producer.properties
max.request.size = 104857600

---consumer.properties
max.partition.fetch.bytes=104857600

-- topic -> payload.max.test
max.message.bytes=104857600

can someone help me if i am missing any other property

rick · 15 September 2021 16:42

@gg24 In looking at the source code, I’ve noticed that this error message has been improved in AK 2.8 to include the configured size in the exception message.

github.com

apache/kafka/blob/afaa12a0968a952dfc601a7485a76399edd8c78f/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L1076


      
              } while (partitionsCount == null || (partition != null && partition >= partitionsCount));
          
          
    return new ClusterAndWaitTime(cluster, elapsed);
          }
          
          
/**
           * Validate that the record size isn't too large
           */
          private void ensureValidRecordSize(int size) {
              if (size > maxRequestSize)
                  throw new RecordTooLargeException("The message is " + size +
                          " bytes when serialized which is larger than " + maxRequestSize + ", which is the value of the " +
                          ProducerConfig.MAX_REQUEST_SIZE_CONFIG + " configuration.");
              if (size > totalMemorySize)
                  throw new RecordTooLargeException("The message is " + size +
                          " bytes when serialized which is larger than the total memory buffer you have configured with the " +
                          ProducerConfig.BUFFER_MEMORY_CONFIG +
                          " configuration.");
          }
          
          
/**

I’m wondering if you could try an upgraded AK client so that you could better verify that your client configuration is taking the desired effect .

Topic		Replies	Views
Kafka: Where can I set max.request.size parameter? Ops	1	8684	30 October 2021
Serializartion of Kafka metrics java.lang.IllegalArgumentException: Message size to large? Confluent Cloud	0	2591	30 August 2022
Kafka Stream producer exception handling Kafka Streams	1	1833	25 April 2023
Kafka record size in between 10-100MB Clients	2	3267	20 July 2022
REST proxy best practices REST Proxy	4	3514	8 March 2021

org.apache.kafka.common.errors.RecordTooLargeException

Related Topics