How to (unit) test repartitioning in KafkaStreams

Kafka Streams
1

So far we used TopologyTestDriver to unit test our KafkaStreams topologies. Unfortunately, it seems to support only a single partition and does not provide any possibilities to extend it in this respect.
Is there a way, with or without TopologyTestDriver, to test repartitioning: events consumed from the same (even single) partition are published to different partitions based on the repartitioning logic?

1 Like
2

You could consider using an embedded Kafka cluster. There are many examples in the streams integration tests:

The basic usage is

    public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);

    @BeforeClass
    public static void startCluster() throws IOException {
        CLUSTER.start();
    }

    @AfterClass
    public static void closeCluster() {
        CLUSTER.stop();
    }

    @Before
    public void before() throws InterruptedException {
        CLUSTER.createTopic(INPUT_TOPIC_NAME, 2, 1);
    }

In the streams configuration, use the embedded kafka as a bootstrap server

        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());

Hope that helps.

3

What exactly are you trying to test with repartitioning that are not already covered?

4

Thank you @lbrutschy and @OneCricketeer

We used TopologyTestDriver so far and were unaware of EmbeddedKafkaCluster, so thanks for the link.

So far I didn’t had enough time to complete such an integration test. First, we’re using Quarkus and RedHat versions and RedHat’s version of org.apache.kafka:kafka-streams was stripped by the test.jar, so I had to use the original version. Then the test I took as a reference used TestUtils, but again RedHat’s version of org.apache.kafka:kafka-clients was stripped by the test.jar and I had to use the original version again. Finally the test was based on Junit4 while we’re using Junit5, which was when I had to switch priorities.

But again, thanks for pointing out EmbeddedKafkaCluster, I totally missed that.