Enabling Product Personalisation Using Apache Kafka, Apache Pinot and Trino [Kafka Summit 2022]

Enabling Product Personalisation Using Apache Kafka, Apache Pinot and Trino
Date : April 25, 2022
Time : 4:00 PM - 4:45 PM BST

Speakers:

  • Stuart Coleman, Head of Product and Engineering for Data and Analytics, 10x Banking

Abstract:
Our core banking platform has been built using domain driven design and microservices and whilst this provides many well-known advantages, it also presents some challenges. Data encapsulation results in each application having its own data store and it becomes impossible to query the state of a customer’s relationship in totality to provide the right products. This challenge becomes even harder if we want to personalise products based on aggregate values of a customer’s behaviour over potentially large periods of time.

In this session, we describe how we overcome this problem to enable dynamic charging and rewards based on customer behaviour in a banking scenario. We describe

  • How we guarantee consistency between our event stream and our OLTP databases using the Outbox pattern.
  • The design decisions faced when considering the schema designs in Pinot and how we balanced flexibility and latency using Trino
  • Two patterns for enriching the event stream using Kafka streams and how we dealt with late arriving events and transactions.