Kafka consumer and a web server simultaneously, thread blocking problem in microservice

Clients
1

assumptions: There are microservices behind an api-gateway, they communicate through HTTP synchronously. obviously, each one of those microservices is a web server. now I want my microservice to play as a Kafka producer and “consumer” too. more clearly, my microservice produces events and listens to some topics for other events.

problem: It seems traditionally a Kafka consumer use an infinite loop to poll the messages and stay alive (send heart-beats) my process thread is busy by serving host and cant be locked by an infinite loop. is there any way to somehow “listen” to the topic without block the thread just like pulsar listener or rabbit? I prefer not to separate my web server and consumer-processor due to the technical complexity I have to handle in development. may I use Kafka streams in this case? Is there something wrong with my assumptions?

2

Hi @engr-Eghbali , thanks for the question.

I may not fully understand your objective, nor what you’d like the webserver to do when it encounters a Kafka event from the consumer… However, if you’re webserver is single threaded and that thread is listening and processing requests from http clients, won’t it need to be interrupted to process the kafka event?

In the past I’ve done this sort of thing using standard threading techniques, so a thread is created to poll and process from the kafka consumer, and the web server thread (pool) is processing http client requests. If you intented to update some internal state from the Kafka event which will eventually be served to your http clients, then of course concurrent access to state needs to be properly managed. Not sure I’ve helped you here, but good luck w/ your project!