I am creating Mailer sending system. For which I have a user’s doc in elastic search. So whenever a Mailer is scheduled, we fetch the user list from elasticsearch based on criteria. and then send mail to the users in two steps. Step 1: pick user list from elasticsearch and insert individual userId to Kafka topic.
Step 2: Kafka consumer pick userId, create mail, and send mail.
Now in Step-1, suppose, we are sending mail to 500,000 users, so I 1st get all 500,000 users from elasticsearch and then producing userId packet to Kafka topic using Async Kafka producer(Spring boot) in batch(100 packets). Now when I am producing packets in Kafka topic, I am getting request-timeout for some packets. so I have increased request-timeout to 120 sec, but again when the user list is greater than 700,000 then again getting this request-timeout.
How to resolve this issue and also suppose I my service is killed or Kafka is suddenly unresponsive then how I will recover the user list?
Or i have to send packets to Kafka topic using Sync approach, but this will take around 60 mins for 500,000 packets. that is too much time.