ForeignKey-Join and Materialization

Hi. I’m designing a solution where low latency and constant processing is an important requirement.

If we take a look at primary-key joins I can do sth like:

val tableA = builder.table(topic-1, Consumed.with(/**/), Materialized.with(/**/)

Then, If I set up num.standby.replicas=1, this table will be “replicated” to another node. And if a processing node dies, then a standby replica will be able to take over the task quickly since it has all the state for a given task present locally (replicated by a changelog topic).

However, I’m not sure how it exactly works with foreign key joins and whether materialization of the input table helps anyhow. I read the article on the confluent website about foreign-key joins, but it’s not easy to understand everything entirely since it’s quite complicated, and quite a few technical topics are created in the process.

To depict the join with the code:

val tableA = builder.table(topic-1, Consumed.with(/**/), Materialized.with(/**/)

tableB.join(tableA, foreignKeyExtractor, ValueJoiner, /*..*/)

Does this materialization of the tableA help anyhow with latency? Can I take any measures to allow kafka-streams take-over the task if an instance leaves the consumer group?

Thanks!

Hi @tomekkra,

Materialization occurs for the KTable even if you don’t include it as a parameter in the overloaded StreamBuilder.table method. The difference between including a Materialized or not is exposure for Interactive Queries - not supplying it means it’s not available for querying.
The best thing you can do for availability is to leverage standby tasks by setting num.standby.replicas> 1, which you’re already doing.

HTH,
Bill