Our team designed an architecture that produces up to 100,000 external API requests per second as data to the AWS MSK cluster as follows.
REST API REQUEST to AWS api gateway → AWS EC2 kafka-client ( running kafka-rest-server ) → AWS MSK cluster
In a test that sends about 10,000 requests per second through the distributed load generator, the kafka-rest-server in the EC2(MSK-kafka-client) encountered the following error,:
"Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1848415041-94”.
We tried a solution to increase the KAFKA_HEAP_OPTS to 7G or increase the producer’s thread to 50 but failed.
Is there a solution in this situation? Please give us your opinion. If you have any similar experience, please advise on architecture.