Recording ready to view: SPEAKER Q&A THREAD: 4 November 2023 - Apache Kafka® Scaling & End to End Exactly Once with Flink and Kafka

Community Events
1

Watch the recording and ask questions about this event in this thread!

  • :spiral_calendar: When: 4 November 2023 at 05:00 (UTC)

  • :speaking_head: Speaker(s): Thyaga Raja, Mudassir Razvi, Irtebat Shaukat

  • :notebook_with_decorative_cover: Talk(s):

    • Kafka Scaling
    • End to End Exactly Once with Flink and Kafka

Kafka Scaling

In this session we will cover two topics:

  • Kafka Horizontal scaling with Kafka Rebalance
  • Kafka Storage downsizing

End to End Exactly Once with Flink and Kafka

In this session we will be going over:

  1. Introduction
  • Brief overview of stream processing.
  1. The importance of data correctness and the challenges of ensuring it.
  2. Understanding Delivery Semantics
  • At-most-once, at-least-once, and exactly-once: What do they mean?
  • Why is exactly-once processing hard in distributed systems?
  1. Apache Kafka’s Role
  • Brief overview of Kafka’s durability guarantees
  • Idempotent producer and Transactions in Kafka
  1. Apache Flink’s Guarantees :
  • Flink’s checkpointing mechanism and distributed snapshots
  • TwoPhaseCommitSinkFunction
  1. Putting It All Together: End-to-end Exactly-Once with Flink & Kafka ( Under the hood )
  2. Q&A Session

:vhs: Find more Meetups Meetup Hub
:earth_africa: Join the Community