Enabling Product Personalisation Using Apache Kafka, Apache Pinot and Trino
Date : April 25, 2022
Time : 4:00 PM - 4:45 PM BST
Speakers:
- Stuart Coleman, Head of Product and Engineering for Data and Analytics, 10x Banking
Abstract:
Our core banking platform has been built using domain driven design and microservices and whilst this provides many well-known advantages, it also presents some challenges. Data encapsulation results in each application having its own data store and it becomes impossible to query the state of a customer’s relationship in totality to provide the right products. This challenge becomes even harder if we want to personalise products based on aggregate values of a customer’s behaviour over potentially large periods of time.
In this session, we describe how we overcome this problem to enable dynamic charging and rewards based on customer behaviour in a banking scenario. We describe
- How we guarantee consistency between our event stream and our OLTP databases using the Outbox pattern.
- The design decisions faced when considering the schema designs in Pinot and how we balanced flexibility and latency using Trino
- Two patterns for enriching the event stream using Kafka streams and how we dealt with late arriving events and transactions.