Split topology creation from presentation of resulting topics

Kafka Streams
1

I’m trying to figure out if/how I can split the processing/creation of a stream topology from the presentation layer. To be more precise, I want to have two separate Java applications, one that deals with the creation of streams, tables, joins, state stores, etc., and eventually writes to topics, and another Java application, that connects to the same Kafka cluster only to expose the data written by the first application. It seems that I cannot access/query the state stores in the second application without re-creating the entire Topology from the first application. Are there any resources that you can point me to in order to understand how I can achieve this or what am I doing wrong?

2

Hi,

If I’m understanding your question correctly it seems you want to use the interactive queries feature of Kafka Streams.

Here are a couple of links to get you started in the right direction:

  1. Interactive Queries section from the Kafka Streams course
  2. Kafka Streams Interactive Queries docs
3

Hello bbejeck,

Thank you for your reply. The question is not about Interactive Queries, it is about how I can set up a microservice to ONLY read by using interactive queries, without re-creating the entire topology that saved the data in the resulting topics.

For e.g., Microservice 1 does something along these lines:

table = builder.stream(inputTopic, Consumed.with(...)).map(...).groupByKey().aggregate(initializer, aggregator, Materialized.with(storeName,...)

Also in Microservice 1:

table.toStream().to(outputTopic, Produced.with(...))

Now, I want to write a different application that would just query the state store defined in microservice 1 as “storeName” and pass that data to a REST controller.

So in Microservice 2, I am doing something along the lines of:

storeQueryParameters =
                StoreQueryParameters.fromNameAndType(
                       storeName,
                        QueryableStoreTypes.<String, byte[]>windowStore());
ReadOnlyWindowStore<String, byte[]> windowStore = streams.store(storeQueryParameters);

(obs. the initial table in microservice 1 was actually a windowed aggregation, but that should not impact the overall description of the issue)

Now, in microservice 2, if my only definition of a topology is something like this:

StreamsBuilder builder = new StreamsBuilder();
Topology topology = builder.build();
streams = new KafkaStreams(topology, props);

The application would not start. I can add something along the lines of

StreamsBuilder builder = new StreamsBuilder();
builder.stream(topicName).print();
Topology topology = builder.build();
streams = new KafkaStreams(topology, props);

And this would start, but the code trying to query the state store would fail, saying that the state store does not exist. If I duplicate the code in Microservice 1 that creates the materialized view of the topic then everything works fine. My intent was to separate the stream composition logic from the “presentation” of the results. (which I tried to query using the InteractiveQuery functionality).

I hope things are more clear now. Let me know if you have questions.

4

Hi superand,

Thanks for the clarification. Off the top of my head, I don’t think this is going to be possible. Did you give the application on microservice 2 the same application-id? But again, I don’t think this will work. There’s a newly accepted KIP for Sharable State Stores that may work for you and It should be available in the 3.3 release.
HTH,
Bill

1 Like
5

Thank you for pointing this out. The KIP seems promising I’ll keep an eye for it. For now we will replicate the topology but will surely use the new feature in 3.3.

Thank you for your help,
Andrei.

6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.