Hi everyone,
I wanted to share something we ran into, in case it saves someone else a few days of investigation.
Setup
- .NET 10 worker in Kubernetes, Debian-based image (glibc)
- Confluent.Kafka 2.5.3 (bundled librdkafka)
- The worker creates one consumer per partition (manually assigned), across several topics, plus a few producers, which comes to roughly 25–30 librdkafka client instances in a single process
- Low throughput: under 100 msg/s at peak
Symptom
Pod memory grew slowly and steadily, by about 100 MB/day, until the container was OOM-killed and restarted. The cycle repeated every few days. Working set grew too, so it wasn’t page cache.
Diagnosis
dotnet-counters showed the managed side was tiny:
- GC heap: ~28 MB
- GC committed memory: ~55 MB
- Container working set: ~690 MB and growing
So the growth was entirely native. Our explanation: each librdkafka client runs several threads (main thread, one per broker, internal threads), so we had well over a hundred native threads allocating and freeing buffers constantly. glibc’s malloc creates up to 8 × CPU cores arenas, counting the node’s cores rather than the container’s CPU limit. Memory freed inside those arenas gets fragmented and is rarely returned to the OS.
Fix
We added ENV MALLOC_ARENA_MAX=2 to the Dockerfile.
Results
| Before | After | |
|---|---|---|
| Main worker | ~725 MB at 5 h uptime, +12 MB/h | ~440 MB, flat |
| Second worker | ~480 MB | ~240 MB |
After 7 days memory is still stable. CPU and consumer lag are unchanged.
Questions for the community
- Has anyone else seen this with many client instances per process?
- Would it make sense to mention
MALLOC_ARENA_MAXin the .NET client docs, or in the librdkafka docs for containerised deployments? - Is one consumer per partition an anti-pattern here? Should we use a single consumer per topic with all partitions assigned?
Thanks!