Kafka 之 KRaft —— ZooKeeper 到 KRaft 的迁移

目录

一. 前言

二. ZooKeeper 到 KRaft 的迁移（ZooKeeper to KRaft Migration）

2.1. 专业术语（Terminology）

2.2 准备迁移（Preparing for migration）

2.3. 提供 KRaft 控制器选举

2.4. 在 Broker 上启用迁移（Enabling the migration on the brokers）

2.5. 将 Broker 迁移到 KRaft（Migrating brokers to KRaft）

2.6. 正在完成迁移（Finalizing the migration）

---

一. 前言

    Kafka 在 3.X 版本后内置了 KRaft 用来替代 ZooKeeper 管理集群，但是在升级的过程中发现，许多升级的文档都是只有新部署安装 KRaft 版本，而没有涉及到数据迁移相关的资料，这样如果直接变更的话，会导致 Kakfa 中的数据全部丢失，这在客户的生产环境中是不可接受的，所以本文把官网的迁移方案作了一下翻译。

二. ZooKeeper 到 KRaft 的迁移（ZooKeeper to KRaft Migration）

原文引用：ZooKeeper to KRaft migration is considered an Early Access feature and is not recommended for production clusters.

The following features are not yet supported for ZK to KRaft migrations:

• Downgrading to ZooKeeper mode during or after the migration
• Other features not yet supported in KRaft

Please report issues with ZooKeeper to KRaft migration using the project JIRA and the "kraft" component.

ZooKeeper 到 KRaft 的迁移被认为是早期访问功能，不建议用于生产集群。

ZK 到 KRaft 的迁移还不支持以下功能：

• 迁移期间或迁移后降级到 ZooKeeper 模式。
• KRaft 中尚未支持的其他功能。

请使用 project JIRA 和“KRaft”组件报告 ZooKeeper 到 KRaft 迁移的问题。

2.1. 专业术语（Terminology）

原文引用：We use the term "migration" here to refer to the process of changing a Kafka cluster's metadata system from ZooKeeper to KRaft and migrating existing metadata. An "upgrade" refers to installing a newer version of Kafka. It is not recommended to upgrade the software at the same time as performing a metadata migration.

    我们在这里使用术语“迁移”是指将 Kafka 集群的元数据系统从 ZooKeeper 更改为 KRaft 并迁移现有元数据的过程。“升级”是指安装 Kafka 的新版本。不建议在执行元数据迁移的同时升级软件。

原文引用：We also use the term "ZK mode" to refer to Kafka brokers which are using ZooKeeper as their metadata system. "KRaft mode" refers Kafka brokers which are using a KRaft controller quorum as their metadata system.

    我们还使用术语“ZK 模式”来指代使用 ZooKeeper 作为元数据系统的 Kafka Broker。“KRaft 模式”是指使用 KRaft 控制器选举作为元数据系统的 Kafka Broker。

2.2 准备迁移（Preparing for migration）

原文引用：Before beginning the migration, the Kafka brokers must be upgraded to software version 3.5.0 and have the "inter.broker.protocol.version" configuration set to "3.5". See Upgrading to 3.5.0 for upgrade instructions.

    在开始迁移之前，Kafka Broker 必须升级到软件版本 3.5.0，并将“inter.broker.procol.version”配置设置为“3.5”。有关升级说明，请参阅 Upgrading to 3.5.0。

原文引用：It is recommended to enable TRACE level logging for the migration components while the migration is active. This can be done by adding the following log4j configuration to each KRaft controller's "log4j.properties" file.

    建议在迁移处于活动状态时为迁移组件启用 TRACE 级别的日志记录。这可以通过将以下 log4j配置添加到每个 KRaft 控制器的“log4j.properties”文件中来实现。

log4j.logger.org.apache.kafka.metadata.migration=TRACE

原文引用：It is generally useful to enable DEBUG logging on the KRaft controllers and the ZK brokers during the migration.

    在迁移过程中，在 KRaft 控制器和 ZK Broker 上启用 DEBUG 日志记录通常很有用。

2.3. 提供 KRaft 控制器选举

原文引用：Two things are needed before the migration can begin. First, the brokers must be configured to support the migration and second, a KRaft controller quorum must be deployed. The KRaft controllers should be provisioned with the same cluster ID as the existing Kafka cluster. This can be found by examining one of the "meta.properties" files in the data directories of the brokers, or by running the following command.

    在迁移开始之前，需要做两件事。首先，必须将 Broker 配置为支持迁移，其次，必须部署 KRaft 控制器选举。KRaft 控制器应配备与现有 Kafka 集群相同的集群 ID。这可以通过检查 Broker 的数据目录中的一个“meta.properties”文件或运行以下命令来找到。

./bin/zookeeper-shell.sh localhost:2181 get /cluster/id

原文引用：The KRaft controller quorum should also be provisioned with the latest metadata.version. This is done automatically when you format the node with the kafka-storage.sh tool. For further instructions on KRaft deployment, please refer to the above documentation.

    KRaft 控制器选举也应配备最新的 metadata.version。使用 kafka-storage.sh 工具格式化节点时，会自动执行此操作。有关 KRaft 部署的更多说明，请参阅上述文档。

原文引用：In addition to the standard KRaft configuration, the KRaft controllers will need to enable support for the migration as well as provide ZooKeeper connection configuration.

Here is a sample config for a KRaft controller that is ready for migration:

除了标准的 KRaft 配置外，KRaft 控制器还需要支持迁移，并提供 ZooKeeper 连接配置。

以下是准备迁移的 KRaft 控制器的配置示例：

# Sample KRaft cluster controller.properties listening on 9093
process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093

# Enable the migration
zookeeper.metadata.migration.enable=true

# ZooKeeper client configuration
zookeeper.connect=localhost:2181

# Other configs ...

原文引用：Note: The KRaft cluster node.id values must be different from any existing ZK broker broker.id. In KRaft-mode, the brokers and controllers share the same Node ID namespace.

    注意：KRaft 集群 node.id 值必须不同于任何现有的 ZK Broker 的 broker.id。在 KRaft 模式下，Broker 和控制器共享相同的 Node ID 命名空间。

2.4. 在 Broker 上启用迁移（Enabling the migration on the brokers）

原文引用：Once the KRaft controller quorum has been started, the brokers will need to be reconfigured and restarted. Brokers may be restarted in a rolling fashion to avoid impacting cluster availability. Each broker requires the following configuration to communicate with the KRaft controllers and to enable the migration.

    一旦 KRaft 控制器选举启动，将需要重新配置和重新启动 Broker。可以以滚动方式重新启动Broker，以避免影响集群可用性。每个 Broker 都需要以下配置才能与 KRaft 控制器通信并启用迁移。 

原文引用：

• controller.quorum.voters
• controller.listener.names
• The controller.listener.name should also be added to listener.security.property.map
• zookeeper.metadata.migration.enable

Here is a sample config for a broker that is ready for migration:

• controller.quorum.voters
• controller.listener.names
• controller.listener.name 也应添加到 listener.security.properties.map
• zookeeper.metadata.migration.enable

以下是一个已准备好进行迁移的代理的示例配置：

# Sample ZK broker server.properties listening on 9092
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

# Set the IBP
inter.broker.protocol.version=3.5

# Enable the migration
zookeeper.metadata.migration.enable=true

# ZooKeeper client configuration
zookeeper.connect=localhost:2181

# KRaft controller quorum configuration
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER

原文引用：Note: Once the final ZK broker has been restarted with the necessary configuration, the migration will automatically begin. When the migration is complete, an INFO level log can be observed on the active controller:

    注意：一旦使用必要的配置重新启动了最终的 ZK Broker，迁移就会自动开始。迁移完成后，可以在活动控制器上观察到 INFO 级别的日志：

Completed migration of metadata from Zookeeper to KRaft

2.5. 将 Broker 迁移到 KRaft（Migrating brokers to KRaft）

原文引用：Once the KRaft controller completes the metadata migration, the brokers will still be running in ZK mode. While the KRaft controller is in migration mode, it will continue sending controller RPCs to the ZK mode brokers. This includes RPCs like UpdateMetadata and LeaderAndIsr.

    一旦 KRaft 控制器完成元数据迁移，Broker 程序仍将以 ZK 模式运行。当 KRaft 控制器处于迁移模式时，它将继续向 ZK 模式 Broker 发送控制器 RPC。这包括 UpdateMetadata 和LeaderAndIsr 等 RPC。

原文引用：To migrate the brokers to KRaft, they simply need to be reconfigured as KRaft brokers and restarted. Using the above broker configuration as an example, we would replace the broker.id with node.id and add process.roles=broker. It is important that the broker maintain the same Broker/Node ID when it is restarted. The zookeeper configurations should be removed at this point. 

    要将 Broker 迁移到 KRaft，只需将它们重新配置为 KRaft Broker 并重新启动即可。以上面的Broker 配置为例，我们将使用 node.id 替换 broker.id，并添加 process.proles=broker。重要的是，Broker 在重新启动时保持相同的 Broker/Node ID。此时应移除 ZooKeeper 配置。

# Sample KRaft broker server.properties listening on 9092
process.roles=broker
node.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://localhost:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT

# Don't set the IBP, KRaft uses "metadata.version" feature flag
# inter.broker.protocol.version=3.5

# Remove the migration enabled flag
# zookeeper.metadata.migration.enable=true

# Remove ZooKeeper client configuration
# zookeeper.connect=localhost:2181

# Keep the KRaft controller quorum configuration
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER

原文引用：Each broker is restarted with a KRaft configuration until the entire cluster is running in KRaft mode.

每个 Broker 都使用 KRaft 配置重新启动，直到整个集群都在 KRaft 模式下运行。

2.6. 正在完成迁移（Finalizing the migration）

原文引用：Once all brokers have been restarted in KRaft mode, the last step to finalize the migration is to take the KRaft controllers out of migration mode. This is done by removing the "zookeeper.metadata.migration.enable" property from each of their configs and restarting them one at a time.

    在 KRaft 模式下重新启动所有 Broker 后，完成迁移的最后一步是使 KRaft 控制器退出迁移模式。这是通过从每个配置中删除“zookeeper.metadata.migration.enable”属性并一次重新启动一个来完成的。

# Sample KRaft cluster controller.properties listening on 9093
process.roles=controller
node.id=3000
controller.quorum.voters=3000@localhost:9093
controller.listener.names=CONTROLLER
listeners=CONTROLLER://:9093

# Disable the migration
# zookeeper.metadata.migration.enable=true

# Remove ZooKeeper client configuration
# zookeeper.connect=localhost:2181

# Other configs ...