Trials, Tribulations, and Triumphs: Migrating from Self-Managed Kafka to Managed Kafka in the Cloud [Kafka Summit 2022]

Trials, Tribulations, and Triumphs: Migrating from Self-Managed Kafka to Managed Kafka in the Cloud
Date : April 26, 2022
Time : 2:00 PM - 2:45 PM BST

Speakers:

  • Adrian Sibilla, Senior Engineering Manager - Data Engineering, Compare the Market
  • Dewi Rees, Senior Engineering Manager - Data Engineering, Compare the Market

Abstract:
Compare the Market has long used Kafka as part of its event data streaming architecture. Whilst Kafka proved a reliable and scalable solution, hosting it in AWS and managing it ourselves came with its own challenges. These challenges included administration of the platform and security, not to mention handling incidents and needing highly skilled talent in the teams to do all of this - and were all contributing factors in driving the decision to migrate.

After much discovery, we landed on ​​Confluent Cloud as our provider of choice to replace our existing Self-Managed Kafka infrastructure. Attend this session to learn about how we went about the migration and issues faced and how this will power our next generation data platform. We will discuss how we overcame the following challenges and more:

  • Identifying a method of replicating topics between platforms
  • Reproducing our existing infrastructure as code with the Confluent Platform and Terraform
  • Coordinating the migration of 100+ consumers and producers, used across the business