Improving Fault Tolerance and Scaling Out in Kafka Streams
Date : April 26, 2022
Time : 3:00 PM - 3:45 PM BST
Speakers:
- Bill Bejeck, DevX, Confluent
Abstract:
Kafka Streams is the popular stream processing component of Apache Kafka®. One of its best features is stateful operations. Kafka Streams works hard to ensure stateful operations can scale horizontally and survive failures, but doing so takes time. Kafka Streams offers the concept of “standby-tasks,” allowing for near-zero downtime failover, but surprisingly this feature still isn’t widely used. The could be for various reasons, from lack of awareness to needing additional resources.
This presentation will cover how standby tasks work and how they’re enabled. Additionally, I’ll cover the work done in KIP-441 that enables faster scaling out for stateful tasks and provides more balanced stateful assignments. I’ll also dive into the consumer rebalance protocol improvements that enable KIP-441 to be effective.
Attendees of this presentation will walk away understanding how and when to use standby tasks, leverage the improvements from KIP-441, and have a deeper understanding of how Kafka Streams works with state.