Kafka for person image crops (10–100 KB): strict no-disk persistence requirement

Hi everyone,

We’re designing an on-premises video analytics system (~200 camera views, NVIDIA GPUs) and evaluating Kafka for transporting person image crops between services.

Context

  • Payloads: JPEG or raw person crops (~10–100 KB), potentially a few hundred per second per server.

  • Privacy requirement: person images must never be persisted to disk and must be deleted promptly when a person leaves the site.

  • Alternative: keep crops in GPU memory for local Re-ID inference and publish only metadata and 384-D embeddings (~1.5 KB each) through Kafka.

Questions

  1. Memory-only operation: Is there a supported production configuration that guarantees Kafka record payloads never reach persistent storage? Specifically, is using tmpfs for log.dirs supported, and what other components or OS mechanisms (KRaft metadata, swap, crash dumps, etc.) would need to be considered?

  2. Record deletion: Can Kafka guarantee deletion of an individual record within a defined time after a deletion request? Would log compaction and tombstones help, or would the record remain until segment cleanup?

  3. Architecture: At this payload size and rate, would you recommend transporting crops through Kafka, or keeping image processing local and publishing only metadata and embeddings? Would a claim-check pattern offer any advantage under a strict no-disk requirement?

  4. Production guidance: Are there documented deployments or recommended practices for Kafka under these privacy constraints?

Our preliminary assessment is that Kafka fits well for metadata and events, but may not be suitable for image payloads requiring strict non-persistence and prompt deletion.

We’d appreciate any corrections, production experience, or relevant documentation.

Thanks!

Kafka is a poor fit for the crop payloads under a strict no-disk rule. Brokers are designed to write records to log segments; a RAM-backed broker directory may reduce disk writes but is not a supported production guarantee, and you still need to control swap, core dumps, page cache and crash artifacts. Kafka also cannot delete an individual record on request; compaction and tombstones affect later cleanup, not prompt erasure. Keep crops in GPU/local memory with TTL and access controls, publish metadata and embeddings through Kafka, and use a claim-check that points to that ephemeral local store—not to Kafka-held images.

Thanks, James. This aligns with our preliminary assessment. We plan to keep person crops in GPU or local memory, with controlled expiration and deletion, and disable swap and core dumps on the processing hosts. Only metadata and embeddings would go through the message broker.

If a remote service needs an image, we may consider a claim-check pattern using temporary in-memory storage.

One remaining concern is appearance embeddings. Under our requirements, these are also treated as personal data, so retention and deletion matter even though each 384-D FP32 embedding is only ~1.5 KB.

A few follow-up questions:

  1. Kafka retention: Is combining a short retention.ms with a small segment.ms the recommended way to minimize how long embeddings remain on brokers? Could replication, tiered storage, or other Kafka components retain copies longer?

  2. Crypto-shredding: Is encrypting messages with short-lived keys and deleting those keys a practical, established approach for this requirement?

  3. NATS vs. Kafka: Since we only need to transport metadata, embeddings, and events, we’re also considering Core NATS for messages that don’t require replay, and JetStream with memory storage and a short MaxAge for messages that do.

Would this be a reasonable approach, or are there technical reasons to prefer Kafka for this workload?

Thanks again for your guidance!

crypto-shredding is the practical one here in my experience, a per-person key you can delete makes the stored embedding unreadable even though the ciphertext hangs around until the segment expires. short retention plus small segments helps with timing but kafka still deletes whole segments, not individual records, so id not rely on it alone for a hard deletion deadline.