Kafka 集群数据备份 MirrorMaker 详解

什么是 MirrorMaker？

MirrorMaker是Kafka附带的一个用于在Kafka集群之间制作镜像数据的工具。该工具主要动作就是从源集群中消费并生产到目标群集。

一个集群可以启动多个MirrorMaker配置到多个集群

 

运行 MirrorMaker方法

kafka-mirror-maker.sh --consumer.config ./consumer.properties --producer.config ./producer.properties --num.streams 8 --whitelist ".*"

实际操作一遍

consumer.properties文件

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat consumer.properties
bootstrap.servers=192.168.3.194:9092
group.id=mirrormaker
auto.offset.reset=earliest

producer.properties文件

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat producer.properties
bootstrap.servers=192.168.2.123:9092

 运行mirror-maker

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat run-mirrormaker.sh
kafka-mirror-maker.sh --consumer.config ./consumer.properties --producer.config ./producer.properties --offset.commit.interval.ms 5000 --num.streams 2 --whitelist "test0428"

参数说明：

--consumer.config 消费者的配置文件（要消费的集群）

--producer.config 指定生产配置文件（要发送到的目标集群）

--whitelist 要同步的topic白名单，可以匹配正则，也可以指定具体topic

--offset.commit.interval.ms 消费端提交offset时间间隔

--num.streams  MirrorMaker 要创建多少个 KafkaConsumer 实例

更多可以使用 --help查看

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# kafka-mirror-maker.sh --help
This tool helps to continuously copy data between two Kafka clusters.
Option                                   Description
------                                   -----------
--abort.on.send.failure <String: Stop    Configure the mirror maker to exit on
  the entire mirror maker when a send      a failed send. (default: true)
  failure occurs>
--consumer.config <String: config file>  Embedded consumer config for consuming
                                           from the source cluster.
--consumer.rebalance.listener <String:   The consumer rebalance listener to use
  A custom rebalance listener of type      for mirror maker consumer.
  ConsumerRebalanceListener>
--help                                   Print usage information.
--message.handler <String: A custom      Message handler which will process
  message handler of type                  every record in-between consumer and
  MirrorMakerMessageHandler>               producer.
--message.handler.args <String:          Arguments used by custom message
  Arguments passed to message handler      handler for mirror maker.
  constructor.>
--new.consumer                           DEPRECATED Use new consumer in mirror
                                           maker (this is the default so this
                                           option will be removed in a future
                                           version).
--num.streams <Integer: Number of        Number of consumption streams.
  threads>                                 (default: 1)
--offset.commit.interval.ms <Integer:    Offset commit interval in ms.
  offset commit interval in                (default: 60000)
  millisecond>
--producer.config <String: config file>  Embedded producer config.
--rebalance.listener.args <String:       Arguments used by custom rebalance
  Arguments passed to custom rebalance     listener for mirror maker consumer.
  listener constructor as a string.>
--version                                Display Kafka version.
--whitelist <String: Java regex          Whitelist of topics to mirror.
  (String)>

 

运行后会提示这么一句话

WARNING: The default partition assignment strategy of the mirror maker will change from 'range' to 'roundrobin' in an upcoming release (so that better load balancing can be achieved). If you prefer to make this switch in advance of that release add the following to the corresponding config: 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor'

 主要是说是MirrorMaker使用的消费策略是 Range ，以后可能改成 “轮训策略” ，我们可以手动指定“轮询策略”

在Consuemr.properties中设置即可

partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

 

 

此时我们已经运行成功，直接去查看是否可以消费成功即可。

注意：同步前最后将目标集群的Topic创建好，否则会使用Broker默认配置