Custom names for intermediate FK-Join topics

Kafka Streams
1

From the discussion Understanding internal topics we’ve learned the purpose of the internal FK-Join topics(Unfortunately I cannot comment on that post as it’s closed, therefore creating a new post).

Is there a way we can give those topics a custom name/identifier to make them relatable to the input topic? Or suffix/prefix it with something that let’s us know to which input topic this automatically generated topic relates to?

As mentioned in the above linked post, the payloads of those topics aren’t really readable, therefore it’s almost impossible to know which input topic given FK join the belongs to.

Right now we have for example quite a few of them and we have no idea which input topic they belong to:

image
image463×564 10.7 KB

Thanks in advance

2

You can pass a TableJoined object into KTable.join(...) to influence names for these internal topics.

3

Hey @mjsax, thanks for the answer. Great to hear about TableJoined parameter, discovered https://github.com/apache/kafka/pull/11368 in the meantime too.

Can you confirm that the usage of TableJoinedwould produce the following topics? (dug a bit into the code to figure out how it’s implemented).

So if I previously had:

tableA.join(
    tableB, 
    keyExtractor, 
    valueJoiner, 
    Materialized.as("")
);

generating the following fk-join topics:

<application.id>-KTABLE-FK-JOIN-SUBSCRIPTION-REGISTRATION-00000000XX-topic
<application.id>-KTABLE-FK-JOIN-SUBSCRIPTION-RESPONSE-00000000XX-topic
<application.id>-KTABLE-FK-JOIN-SUBSCRIPTION-STATE-STORE-00000000XX-topic

Which I’m assuming is belonging to one specific join.

And now if I’d do

tableA.join(
    tableB, 
    keyExtractor, 
    valueJoiner, 
    TableJoined.as("tableA-to-tableB-fk-join"),
    Materialized.as("")
);

would I get:

<application.id>-tableA-to-tableB-fk-join-subscription-registration-topic
<application.id>-tableA-to-tableB-fk-join-subscription-response-topic
<application.id>-tableA-to-tableB-fk-join-subscription-state-store-topic

Is that right?

4

Update: I’ve managed to utilize Topology.describe() to check the results. Indeed it works!

1 Like