For GDPR reasons we have many of our long-lived topics controlled by removing records by configuring topics to
compact, delete and producing tombstone records for the relevant keys. Configuration and the mechanics of compaction is clear to me and this setup works in most cases.
However, compaction is never allowed to run on active segments. And regardless of how segment roll is configured it appears that unless a record is written to a partition a new segment will not become active. For topics/partitions with low or perhaps even no traffic this means that compaction will in fact never run and records lingers indefinitely.
Is it in fact impossible to create a new empty active segment? If so I guess we need to setup some sort of monitoring of all topics and generate dummy records where appropriate.
Thankful for any input.