We have a centralized Kafka cluster in the organization I work for.
Most of the data gets written to a single topic (Almost 5 billion events are ingested per day) and then read off by multiple consumers.
Sometimes due to some internal constraints, a subset of these consumers start lagging. This is not an issue for our application as this does not impact any user facing system.
What we have noticed though is this - Anytime a substantial number of consumers start lagging, our write latencies increase as well. Now this is completely counterintuitive given that producers and consumers are isolated from each other.
Is this a known behaviour or this is something unique to our setup ?