Kafka mirror

最新推荐文章于 2024-05-14 22:01:07 发布

胖胖胖胖胖虎

最新推荐文章于 2024-05-14 22:01:07 发布

阅读量1.4k

点赞数

分类专栏： K8S Kafka 文章标签： kafka k8s

本文链接：https://blog.csdn.net/qq_15138049/article/details/122217685

版权

K8S 同时被 2 个专栏收录

102 篇文章 1 订阅

订阅专栏

Kafka

16 篇文章 1 订阅

订阅专栏

准备两个Kafka集群（default | loganalysis）

在这里插入图片描述

配置文件

### loganalysis kafka-0 ###
root@kafka-0:/etc/kafka# ls
connect-console-sink.properties    connect-file-sink.properties    connect-standalone.properties  log4j.properties     server.properties           tools-log4j.properties
connect-console-source.properties  connect-file-source.properties  consumer.properties            producer.properties  source-consumer.properties  trogdor.conf
connect-distributed.properties     connect-log4j.properties        kafka.properties               secrets              target-producer.properties  zookeeper.properties

### 数据源 Kafka ###

root@kafka-0:/etc/kafka# cat source-consumer.properties
bootstrap.servers=kafka-headless:9092
zookeeper.connect=kafka-zookeeper-headless:2181
zookeeper.connection.timeout.ms=1000000
group.id=mirror-maker-group-test
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000

### 目标 Kafka ###

root@kafka-0:/etc/kafka# cat target-producer.properties
bootstrap.servers=loganalysis-kafka-headless.default:9092
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000
root@kafka-0:/etc/kafka#

kafka-mirror

root@kafka-2:/# kafka-mirror-maker
Continuously copy data between two Kafka clusters.
Option                                   Description
------                                   -----------
--abort.on.send.failure <String: Stop    Configure the mirror maker to exit on
  the entire mirror maker when a send      a failed send. (default: true)
  failure occurs>
--consumer.config <String: config file>  Embedded consumer config for consuming
                                           from the source cluster.
--consumer.rebalance.listener <String:   The consumer rebalance listener to use
  A custom rebalance listener of type      for mirror maker consumer.
  ConsumerRebalanceListener>
--help                                   Print this message.
--message.handler <String: A custom      Message handler which will process
  message handler of type                  every record in-between consumer and
  MirrorMakerMessageHandler>               producer.
--message.handler.args <String:          Arguments used by custom message
  Arguments passed to message handler      handler for mirror maker.
  constructor.>
--new.consumer                           DEPRECATED Use new consumer in mirror
                                           maker (this is the default so this
                                           option will be removed in a future
                                           version).
--num.streams <Integer: Number of        Number of consumption streams.
  threads>                                 (default: 1)
--offset.commit.interval.ms <Integer:    Offset commit interval in ms.
  offset commit interval in                (default: 60000)
  millisecond>
--producer.config <String: config file>  Embedded producer config.
--rebalance.listener.args <String:       Arguments used by custom rebalance
  Arguments passed to custom rebalance     listener for mirror maker consumer.
  listener constructor as a string.>
--whitelist <String: Java regex          Whitelist of topics to mirror.
  (String)>
root@kafka-2:/#

root@kafka-0:/etc/kafka# kafka-mirror-maker --consumer.config source-consumer.properties --producer.config target-producer.properties  --whitelist test