Why foreignKeyExtractor does not accept key?

Stream Processing Kafka Streams
1

Hi,

We are quite often using a foreign key join to join data where left side is keyed by <producer_id, product_id> and right side is <producer_id>. (let’s assume that product_id can be repeated between producer_ids).
However, foreignKeyExtractor is only accepting value of the left side which forces us to copy producer_id in the value of the left side. Is there a reason for this design choice?

Ideally, we would be able to custom partition the left side using the producer_id to ensure that both sides are co-partitioned and do a foreign key join without the intermediate topics. I saw KIP-759: Unneeded repartition canceling - Apache Kafka - Apache Software Foundation and the hack in Kafka Streams: a map not repartitioning co-partitioned data - Stack Overflow
but I don’t think either would allow me to do that.

Is there a better way of solving this?

2

Good point. I think it’s just a oversight in the design of FK join. I don’t see any reason why we would need to limit the FK-extractor to only user the value as input.

Can you file a Jira ticket for this?

I cannot think of any other workaround right now, but believe what you do (duplicate the FK key column into the value) is the only way to make it work right now.

3

https://issues.apache.org/jira/browse/KAFKA-17893