Kafka Producer - Do I need to close the producer connection?

Clients
1

Is it okay to have the producer running indefinitely once a connection is established?

There are couple of reasons for asking the question?

  1. What is the implication on the underlying resource/s (leader replica) when there is always an active open connection at all times?
  2. Idempotency only lasts for the lifetime of a producer instance. So restarting a new instance takes away a key feature/guarantee of the Kafka system
  3. What are the suggested best practices when it comes to maintaining the producer connection?
  4. For an application (like click streams that is constantly producing message), is there ever a need to close the producer connection?
  5. If for (4) there is no need to close the producer, can this be made generic to ALL applications and not close the producer connection at all?

Much appreciated.

2

I think it’s mostly a matter of scale, and often it’s fine to keep the connection open. A good practice is to use one producer for each app, and not have a seperate producer for each service for example.
There are exemptions to each rule, like the serialization config might be different, or when the throughput is crazy.

The same thing for closing the producer. I can image scenarios where you might want to close it. For example if you have hundreds of instances collecting iot data. In which case it’s probably more efficient to send them in bursts, sacrificing having the data a bit later for needing enough brokers to keep all the connections open. The same kind of thing would hold when you have hundreds of microservices doing some batch processing.

I would like to hear other opinions, for me at least I think I only really closed producers during integration tests.