There’s a new Streaming Audio episode - check it out!
Jason Gustafson and Colin McCabe, Apache Kafka® developers, discuss all things KIP-500 adoption, the removal of ZooKeeper, and how that’s played out on the frontlines within the event streaming world. A previous episode of Streaming Audio featured both developers on the podcast before the release of Apache Kafka 2.8. Now they’re back to share how everything is working in reality.
The code has been merged and continues to be merged in phases. Both developers talk about the foundational Kafka Improvement Proposals (KIPs), such as Raft implementation and Control 1 KIP-63, as well as node memory retention and security features. The idea going into this new release was to scale (both scaling up and down) as many of the development pipelines as possible into actionable KIPs as the development team continues to figure out new ways to make workloads easier within the 2.8 framework.
The introduction of Kafka 2.8’s APIs has also allowed multi-cloud functionality throughout the entire platform. According to Jason and Colin, making Kafka compatible with all of these microservices in cloud and on-premises was always going to be the biggest challenge. The only way to know about those hurdles was to get into the update for in-depth integration testing.
After the release and removal of ZooKeeper, much of the old reference architecture was exposed through the new APIs and made the migration more challenging. As anticipated, this created the need for the development team to develop new innovative mechanisms for problem-solving this lack of access. 2.8 added the ability for locally cached copy that supplies the metadata without the go-between of ZooKeeper. The update has led to increased performance implications that allow all nodes within a cluster to be shown in the same order.
Jason and Colin plan to continue working on upgrades and making sure those upgrades work smoothly so that the additional phases of the 2.8 rollout will be a complement to cloud pipelines. By the time 3.0 is released, they hope that most anomaly detection efforts will be completed and data migration will occur without issue.
EPISODE LINKS
- KIP-500: Apache Kafka Without ZooKeeper ft. Colin McCabe and Jason Gustafson
- What’s New in Apache Kafka 2.8
- Watch the video version of this podcast
- Join the Confluent Community
- Learn more with Kafka tutorials, resources, and guides at Confluent Developer
- Live demo: Kafka streaming in 10 minutes on Confluent Cloud
- Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)
Listen to the episode