Migrating v old installation (v5.x) to latest

Ops
1

I have 5 brokers running rocky 9 and ‘confluent-kafka-2.11-2.0.1cp8-1’.

The yum repo is ‘https://packages.confluent.io/rpm/5.0’ - which Im guessing is 3 major versions behind. The goal is to land on the latest (v8.x) without too much disruption of existing data.

I’ve tried looking for guidance on doing the upgrade but it seems that the 5→6 path is too old. All the links go to the same Confluent Documentation | Confluent Documentation which doesn’t appear to have any information on this (v dated) upgrade path.

Any chance there’s an archive somewhere?

2

You can download docs for older Confluent Platform versions (7.2 and under) here. 5 –> 8 is a big jump and you might run into a bunch of pain upgrading in place. I’m not sure how painful it’ll be – just a hunch that it’ll be a hassle. Depending on how the process goes, you may want to keep the idea of data migration to new cluster in your back pocket (i.e., spin up a fresh 8.1 cluster and migrate data).

3

I was figuring that it would be like k8 - I’d have to go 5→6→7→8 and possibly to some intermediate steps as well.

4

hey @ebjy

as @dtroiano said, it might be (a bit) painful I would recommend to try the data migration approach with mirrormaker or so.

best, michael

5

Unfortunately this is all metal and not handy cloud vms so I’m limited in migration. I’ll have to upgrade each piece separately to the target version.. test the works.. and repeat with the next version.

My main concern is that I don’t miss any mandatory versions on the way.

6

On second thought.. maybe the mirrormaker approach would (ultimately) be easier. Now just need to figure out how to maintain continuity for connected clients.

7

Got it – probably goes without saying that you’ll want to do a bunch of reading first (release / upgrade notes for all versions along the way).

To be conservative, you might want to imagine what you would have to do if your Kafka cluster winds up hosed during the process, and proactively create a plan involving backup / restore, etc.

8

To be clear: if my repo is pointing at ‘baseurl=https://packages.confluent.io/rpm/5.0’

–and–

$rpm -qa | grep kafka
confluent-kafka-2.11-2.0.1cp8-1.noarch

Am I on confluent v5 or is it much older than that?

9

Looks like you’re on v5 to me. That baseurl corresponds to CP 5. CP major version 5.0.x corresponds to Apache Kafka version 2.0.x, and 2.11 is the only supported Scala version for CP 5, so I think that the 2.11 in the Kafka package is Scala and 2.0.1 is the Apache Kafka version it’s based on.

10

Thanks.

Taking it a few versions at a time I was able to get to 7.7 . After that it looks like there are some breaking changes before 8.1. Need to make sure all the code is up to date.