Hi. I’m designing a solution where low latency and constant processing is an important requirement.
If we take a look at primary-key joins I can do sth like:
val tableA = builder.table(topic-1, Consumed.with(/**/), Materialized.with(/**/)
Then, If I set up num.standby.replicas=1
, this table will be “replicated” to another node. And if a processing node dies, then a standby replica will be able to take over the task quickly since it has all the state for a given task present locally (replicated by a changelog topic).
However, I’m not sure how it exactly works with foreign key joins and whether materialization of the input table helps anyhow. I read the article on the confluent website about foreign-key joins, but it’s not easy to understand everything entirely since it’s quite complicated, and quite a few technical topics are created in the process.
To depict the join with the code:
val tableA = builder.table(topic-1, Consumed.with(/**/), Materialized.with(/**/)
tableB.join(tableA, foreignKeyExtractor, ValueJoiner, /*..*/)
Does this materialization of the tableA help anyhow with latency? Can I take any measures to allow kafka-streams take-over the task if an instance leaves the consumer group?
Thanks!