KIP-382: MirrorMaker 2.0(中英文对照)

• Status（状态）
• Motivation（动机）
• Public Interfaces（公共接口）
• Proposed Changes（改变）
  • Remote Topics, Partitions（远程主题，分区）
  • Aggregation（聚合）
  • Cycle detection（循环检测）
  • Config, ACL Sync（配置，ACL同步）
  • Internal Topics（内部主题）
  • Remote Cluster Utils（远程群集实用程序）
  • MirrorClient（Mirror客户端）
  • Replication Policies and Filters（复制策略和过滤器）
    • Connector Configuration Properties（连接器配置属性）
    • Example Configuration（配置示例）
    • Legacy MirrorMaker Configuration（旧版MirrorMaker配置）
  • MirrorMaker Clusters（MirrorMaker群集）
    • MirrorMaker Configuration Properties（MirrorMaker配置属性）
• Walkthrough: Running MirrorMaker 2.0（演练：运行MirrorMaker 2.0）
  • Running a dedicated MirrorMaker cluster（运行专用的MirrorMaker集群）
  • Running a standalone MirrorMaker connector（运行独立的MirrorMaker连接器）
  • Running MirrorMaker in a Connect cluster（在Connect群集中运行MirrorMaker）
  • Running MirrorMaker in legacy mode（在旧版模式下运行MirrorMaker）
• Compatibility, Deprecation, and Migration Plan（兼容性，弃用和迁移计划）
• Future Work（未来的工作）
• Rejected Alternatives（拒绝的选择）

Status

Current state: Accepted（当前状态：已 接受）

Discussion thread: here（讨论主题：此处）

JIRA:  KAFKA-7500 - MirrorMaker 2.0 (KIP-382) RESOLVED  JIRA： KAFKA-7500 - MirrorMaker 2.0（KIP-382） 解决

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast). 

请将讨论内容保留在邮件列表中，而不要对Wiki进行评论（Wiki讨论很快变得难以处理）。

Motivation（动机）

MirrorMaker has been used for years in large-scale production environments, but not without several problems:

MirrorMaker已经在大型生产环境中使用了多年，但有几个问题：

• Topics are created with default configuration. Often they'll need to be repartitioned manually.
• 主题是使用默认配置创建的。通常，它们需要手动重新分区。
• ACL and configuration changes are not synced across mirrored clusters, making it difficult to manage multiple clusters.
• ACL和配置更改不会在镜像群集之间同步，因此很难管理多个群集。
• Records are repartitioned with DefaultPartitioner. Semantic partitioning may be lost.
• 使用DefaultPartitioner对记录进行重新分区。语义分区可能会丢失。
• Any configuration change means the cluster must be bounced. This includes adding new topics to the whitelist, which may be a frequent operation.
• 任何配置更改都意味着必须将群集反弹（bounced？）。这包括将新主题添加到白名单中，这可能是一项频繁的操作。
• There's no mechanism to migrate producers or consumers between mirrored clusters.
• 没有在镜像集群之间迁移生产者或消费者的机制。
• There's no support for exactly-once delivery. Records may be duplicated during replication.
• 不支持一次精确交付。在复制过程中，记录可能会重复。
• Clusters can't be made mirrors of each other, i.e. no support for active/active pairs.
• 群集不能成为彼此的镜像，即不支持主动/主动对。
• Rebalancing causes latency spikes, which may trigger further rebalances.
• 重新平衡会导致延迟峰值，这可能会触发进一步的重新平衡。

For these reasons, MirrorMaker is insufficient for many use cases, including backup, disaster recovery, and fail-over scenarios. Several other Kafka replication tools have been created to address some of these limitations, but Apache Kafka has no adequate replication strategy to date. Moreover, the lack of a native solution makes it difficult to build generic tooling for multi-cluster environments.

由于这些原因，MirrorMaker无法满足许多用例，包括备份，灾难恢复和故障转移场景。其他几种Kafka复制工具已经解决了其中一些局限性，但是Apache Kafka迄今没有足够的复制策略。而且，由于缺少本机解决方案，因此很难为多集群环境构建通用工具。

I propose to replace MirrorMaker with a new multi-cluster, cross-data-center replication engine based on the Connect framework, MirrorMaker 2.0 (MM2). The new engine will be fundamentally different from the legacy MirrorMaker in many ways, but will provide a drop-in replacement for existing deployments.

我建议用MirrorMaker 2.0（MM2）来替换MirrorMaker，MirrorMaker 2.0（MM2基于Connect框架，是一个新的多集群，跨数据中心复制引擎。新引擎将在许多方面与旧版MirrorMaker根本不同，但是它将为现有部署提供直接替代。

 

Highlights of the design include:

设计的重点包括：

• Leverages the Kafka Connect framework and ecosystem.
• 利用Kafka Connect框架和生态系统。
• Includes both source and sink connectors.
• 包括源连接器和接收器连接器。
• Includes a high-level driver that manages connectors in a dedicated cluster.
• 包括管理专用集群中连接器的高级驱动程序。
• Detects new topics, partitions.
• 检测新主题，分区。
• Automatically syncs topic configuration between clusters.
• 自动在集群之间同步主题配置。
• Manages downstream topic ACL.
• 管理下游主题ACL。
• Supports "active/active" cluster pairs, as well as any number of active clusters.
• 支持“活动/活动”群集对以及任意数量的活动群集。
• Supports cross-datacenter replication, aggregation, and other complex topologies.
• 支持跨数据中心复制，聚合和其他复杂的拓扑。
• Provides new metrics including end-to-end replication latency across multiple data centers/clusters.
• 提供新的指标，包括跨多个数据中心/集群的端到端复制延迟。
• Emits offsets required to migrate consumers between clusters.
• 发出在群集之间迁移使用者的偏移量。
• Tooling for offset translation.
• 迁移偏移量的工具。
• MirrorMaker-compatible legacy mode.
• 兼容MirrorMaker旧模式。
• No rebalancing.
• 没有重新平衡。

Public Interfaces（公共接口）

This KIP subsumes the small interface change proposed in KIP-416: Notify SourceTask of ACK'd offsets, metadata

此KIP包含了KIP-416中提议的小接口更改 ：通知ACK偏移，元数据的SourceTask

New classes and interfaces include:

新的类和接口包括：

• MirrorSourceConnector, MirrorSinkConnector, MirrorSourceTask, MirrorSinkTask classes.
• MirrorCheckpointConnector, MirrorCheckpointTask.
• MirrorHeartbeatConnector, MirrorHeartbeatTask.
• MirrorConnectorConfig, MirrorTaskConfig classes.
• ReplicationPolicy interface. DefaultReplicationPolicy and LegacyReplicationPolicy classes.
• Heartbeat, checkpoint, offset sync topics and associated schemas.
• 心跳，检查点，偏移同步主题和关联的架构。
• RemoteClusterUtils and MirrorClient classes for querying remote cluster reachability and lag, and for translating consumer offsets between clusters.
• RemoteClusterUtils 和MirrorClient类用于查询远程集群的可达性和滞后性，以及转换集群之间的使用者偏移量。
• MirrorMaker driver class with main entry point for running MM2 cluster nodes.
• 带有主要入口点的MirrorMaker驱动程序类，用于运行MM2群集节点。
• MirrorMakerConfig used by MirrorMaker driver.
• MirrorMaker驱动程序使用MirrorMakerConfig。
• HeartbeatMessageFormatter, CheckpointMessageFormatter
• ./bin/connect-mirror-maker.sh and ./config/mirror-maker.properties sample configuration.
• ./bin/connect-mirror-maker.sh和./config/mirror-maker.properties示例配置。

New metrics include:新指标包括：

• replication-latency-ms(-avg/-min/-max): timespan between each record's timestamp and downstream ACK
• 复制延迟毫秒（-avg / -min / -max）：每条记录的时间戳和下游ACK之间的时间跨度
• record-bytes(-avg/-min/-max): size of each record being replicated
• 记录字节数（-avg / -min / -max）：每个要复制的记录的大小
• record-age-ms(-avg/-min/-max): age of each record when consumed
• record-age-ms（-avg / -min / -max）：每条记录的使用期限
• record-count: number of records replicated
• record-count：复制的记录数
• checkpoint-latency-ms(-avg/-min/-max): timestamp between consumer group commit and downstream checkpoint ACK
• checkpoint-latency-ms（-avg / -min / -max）：consumer group提交和下游检查点ACK之间的时间戳记

 The mbean name for these metrics will be( 这些指标的mbean名称为): kafka.mirror.connect:type=MirrorSourceConnect,target=([.\w]+),topic=([.\w]+),partition=([.\d]+) and（和 ） kafka.mirror.connect:type=MirrorCheckpointConnector,target=([.\w]+),source=([.\w]+),group=([.\w]+)

Proposed Changes（变更）

Remote Topics, Partitions（远程主题，分区）

This design introduces the concept of remote topics, which are replicated topics referencing a source cluster via a naming convention, e.g. "us-west.topic1", where topic1 is a source topic and us-west is a source cluster alias. Any partitions in a remote topic are remote partitions and refer to the same partitions in the source topic. Specifically:

此设计引入了远程主题的概念，该主题是通过命名约定（例如“ us-west.topic1”）引用源集群的复制主题，其中topic1是源主题，而us-west是源集群别名。远程主题中的任何分区都是远程分区，并且在源主题中引用相同的分区。特别：

• Partitioning and order of records is preserved between source and remote topics.
• 记录的分区和顺序保留在源主题和远程主题之间。
• Remote topics must have the same number of partitions as their source topics.
• 远程主题必须与源主题具有相同数量的分区。
• A remote topic has a single source topic.
• 远程主题只有一个源主题。
• A remote partition has a single source partition.
• 远程分区具有单个源分区。
• Remote partition i is a replica of source partition i.
• 远程分区i是源分区i的副本。

MM2 replicates topics from a source cluster to corresponding remote topics in the destination cluster. In this way, replication never results in merged or out-of-order partitions. Moreover, the topic renaming policy enables "active/active" replication by default:

MM2将主题从源集群复制到目标集群中的相应远程主题。这样，复制永远不会导致合并或混乱的分区。此外，主题重命名策略默认情况下会启用“主动/主动”复制：

 

"active/active" replication(“主动/主动”复制)

This is not supported by legacy MirrorMaker out-of-the-box (without custom handlers) -- records would be replicated back and forth indefinitely, and the topics in either cluster would be merged inconsistently between clusters. The remote naming convention avoids these problems by keeping local and remote records in separate partitions. In either cluster above, the normal topic1 contains only locally produced records. Likewise, B.topic1 contains only records produced in cluster B.

旧版MirrorMaker不支持开箱即用（不带自定义处理程序）-记录将无限期地来回复制，并且两个群集中的主题将在多个群集之间不一致地合并。远程命名约定通过将本地和远程记录保存在隔离的分区中避免了这些问题。在以上任一集群中，常规topic1仅包含本地生成的记录。同样，B.topic1仅包含群集B中产生的记录。

To consume local records only, a consumer subscribes to topic1 as normal. To consume records that have been replicated from a remote cluster, consumers subscribe to a remote topic, e.g. B.topic1.

仅使用本地记录，使用者通常订阅topic1。要使用已从远程群集复制的记录，使用者可以订阅一个远程主题，例如B.topic1。

This convention extends to any number of clusters.

此约定扩展到任意数量的群集。

Aggregation（聚合）

Downstream consumers can aggregate across multiple clusters by subscribing to the local topic and all corresponding remote topics, e.g. topic1, us-west.topic1, us-east.topic1… or by using a regex subscription.

通过订阅本地主题和所有相应的远程主题（例如topic1，us-west.topic1，us-east.topic1…）或使用正则表达式订阅，下游使用者可以跨多个群集进行聚合。

Topics are never merged or repartitioned within the connector. Instead, merging is left to the consumer. This is in contrast to MirrorMaker, which is often used to merge topics from multiple clusters into a single aggregate cluster.

主题永远不会在连接器内合并或重新分区。相反，合并留给消费者。这与MirrorMaker相反，后者通常用于将多个集群中的主题合并到单个聚合集群中。

Likewise, a consumer can always elect not to aggregate by simply subscribing to the local topic only, e.g. topic1.

同样，消费者始终可以选择只订阅本地主题（例如topic1）来选择不进行汇总。

This approach eliminates the need for special purpose aggregation clusters without losing any power or flexibility.

这种方法消除了对专用聚合群集的需求，而不会失去任何功能或灵活性。

Cycle detection（循环检测）

It is possible to configure two clusters to replicate each other ("active/active"), in which case all records produced to either cluster can be seen by consumers in both clusters. To prevent infinite recursion, topics that already contain "us-west" in the prefix won't be replicated to the us-west cluster.

可以将两个群集配置为彼此复制（“活动/活动”），在这种情况下，两个群集中的使用者都可以看到向任一群集生成的所有记录。为防止无限递归，前缀中已经包含“ us-west”的主题将不会复制到us-west集群。

This rule applies across all topics regardless of topology. A cluster can be replicated to many downstream clusters, which themselves can be replicated, yielding topics like us-west.us-east.topic1 and so on. The same cluster alias will not appear twice in a topic name due to cycle detection.

该规则适用于所有主题，而与拓扑无关。一个集群可以复制到许多下游集群，而下游集群本身也可以复制，产生诸如us-west.us-east.topic1之类的主题。由于周期检测，同一群集别名在主题名称中不会出现两次。

In this way, any topology of clusters is supported, not just DAGs.

这样，不仅支持DAG，还支持任何集群拓扑。

Config, ACL Sync（配置，ACL同步）

MM2 monitors source topics and propagates configuration changes to remote topics. Any missing partitions will be automatically created.

MM2监视源主题并将配置更改传播到远程主题。任何丢失的分区将自动创建。

Remote topics should never be written to, except by the source or sink connector. Producers must only write to local topics (e.g. topic1), which may then be replicated elsewhere. To enforce this policy, MM2 propagates ACL policies to downstream topics using the following rules:

除源或sink连接器外，切勿写入远程主题。生产者只能写本地主题（例如topic1），然后可以复制到其他地方。为了实施此策略，MM2使用以下规则将ACL策略传播到下游主题：

• Principals that can READ the source topic can READ the remote topic.
• 可以读取源主题和远程主题。
• No one can WRITE to the remote topic except MM2.
• 除了MM2，没有人可以写到远程主题。

Internal Topics（内部主题）

MM2 emits a heartbeat topic in each source cluster, which is replicated to demonstrate connectivity through the connectors. Downstream consumers can use this topic to verify that a) the connector is running and b) the corresponding source cluster is available. Heartbeats will get propagated by source and sink connectors s.t. chains like backup.us-west.us-east.heartbeat are possible. Cycle detection prevents infinite recursion.

MM2 在每个源群集中发出一个 心跳 主题，将其复制以演示通过连接器的连接。下游使用者可以使用此主题来验证a）连接器正在运行，并且b）相应的源群集可用。心跳将通过源连接器和接收器连接器传播，可能会出现st链，例如backup.us-west.us-east.heartbeat。循环检测可防止无限递归。

 

heartbeats are replicated between clusters

心跳在群集之间复制

The schema for the heartbeat topic contains:

心跳主题的模式包含：

• target cluster (String): cluster where the heartbeat was emitted
• 目标群集（字符串）：发出心跳的群集
• source cluster (String): source cluster of connector doing the emitting
• 源群集（字符串）：执行发射的连接器的源群集
• timestamp (long): epoch millis when heartbeat was created
• 时间戳记（长）：创建心跳时的纪元毫秒

Additionally, the connectors periodically emit checkpoints in the destination cluster, containing offsets for each consumer group in the source cluster. The connector will periodically query the source cluster for all committed offsets from all consumer groups, filter for those topics being replicated, and emit a message to a topic like us-west.checkpoints.internal in the destination cluster. The message will include the following fields:

此外，连接器会定期 在目标群集中发出 检查点，其中包含源群集中每个使用者组的偏移量。连接器将定期在源集群中查询所有使用者组的所有已提交偏移量，过滤出要复制的主题，并向目标集群中的主题us-west.checkpoints.internal发送消息。该消息将包括以下字段：

 

• consumer group id (String)
• 消费者组ID（字符串）
• topic (String) – includes source cluster prefix
• 主题（字符串）–包括源集群前缀
• partition (int)
• 分区（int）
• upstream offset (int): latest committed offset in source cluster
• 上游偏移量（int）：源集群中最新提交的偏移量
• downstream offset (int): latest committed offset translated to target cluster
• 下游偏移量（整数）：被迁移到目标集群的最新提交偏移量
• metadata (String)
• 元数据（字符串）
• timestamp
• 时间戳记

As with  __consumer_offsets, the checkpoint topic is log-compacted to reflect only the latest offsets across consumer groups, based on a topic-partition-group composite key. The topic will be created automatically by the connector if it doesn't exist.

与__consumer_offsets一样，基于主题-分区-组组合键，检查点主题经过日志压缩以仅反映消费者组之间的最新偏移量。如果该主题不存在，则它将由连接器自动创建。

Finally, an offset sync topic encodes cluster-to-cluster offset mappings for each topic-partition being replicated.

最后， 偏移同步主题为要复制的每个主题分区编码集群到集群的偏移映射。

• topic (String): remote topic name
• topic（字符串）：远程主题名称
• partition (int)
• 分区（int）
• upstream offset (int): an offset in the source cluster
• 上游偏移量（int）：源群集中的偏移量
• downstream offset (int): an equivalent offset in the target cluster
• 下游偏移量（整数）：目标群集中的等效偏移量

The heartbeat, checkpoint, and offset sync records are encoded using Kafka's internal serdes, as with __consumer_offsets. Relevant classes below.

心跳，检查点和偏移同步记录使用Kafka的内部Serdes（与__consumer_offsets一样）进行编码。相关类别如下。

public class Checkpoint {     public static final String TOPIC_KEY = "topic";     public static final String PARTITION_KEY = "partition";     public static final String CONSUMER_GROUP_ID_KEY = "group";     public static final String UPSTREAM_OFFSET_KEY = "upstreamOffset";     public static final String DOWNSTREAM_OFFSET_KEY = "offset";     public static final String METADATA_KEY = "metadata";   --->%---       public Checkpoint(String consumerGroupId, TopicPartition topicPartition, long upstreamOffset,             long downstreamOffset, String metadata) ...       public String consumerGroupId() ...       public TopicPartition topicPartition() ...       public long upstreamOffset() ...       public long downstreamOffset() ...       public String metadata() ...       public OffsetAndMetadata offsetAndMetadata() ...   --->%---   public class Heartbeat {     public static final String SOURCE_CLUSTER_ALIAS_KEY = "sourceClusterAlias";     public static final String TARGET_CLUSTER_ALIAS_KEY = "targetClusterAlias";     public static final String TIMESTAMP_KEY = "timestamp";   --->%---       public Heartbeat(String sourceClusterAlias, String targetClusterAlias, long timestamp) ...       public String sourceClusterAlias() ...       public String targetClusterAlias() ...       public long timestamp() ...   --->%---   public class OffsetSync {     public static final String TOPIC_KEY = "topic";     public static final String PARTITION_KEY = "partition";     public static final String UPSTREAM_OFFSET_KEY = "upstreamOffset";     public static final String DOWNSTREAM_OFFSET_KEY = "offset";   --->%---       public OffsetSync(TopicPartition topicPartition, long upstreamOffset, long downstreamOffset) ...       public TopicPartition topicPartition() ...       public long upstreamOffset() ...       public long downstreamOffset() ...

Remote Cluster Utils（远程群集实用程序）

A utility class RemoteClusterUtils will leverage the internal topics described above to assist in computing reachability, inter-cluster lag, and offset translation. It won't be possible to directly translate any given offset, since not all offsets will be captured in the checkpoint stream. But for a given consumer group, it will be possible to find high water marks that consumers can seek() to. This is useful for inter-cluster consumer migration, failover, etc.

实用程序类 RemoteClusterUtils 将利用上述内部主题来帮助计算可达性，集群间延迟和偏移量转换。因为并非所有偏移量都会在检查点流中捕获，所以它无法直接转换任何给定的偏移量。但是对于给定的消费者群体，将有可能找到消费者可以寻求seek() 的高水位线。这对于集群间消费者迁移，故障转移等很有用。

The interface is as follows:

接口如下:

// Calculates the number of hops between a client and an upstream Kafka cluster based on replication heartbeats. // If the given cluster is not reachable, returns -1. int replicationHops(Map<String, Object> properties, String upstreamClusterAlias)   // Find all heartbeat topics, e.g. A.B.heartbeat, visible from the client. Set<String> heartbeatTopics(Map<String, Object> properties)   // Find all checkpoint topics, e.g. A.checkpoint.internal, visible from the client. Set<String> checkpointTopics(Map<String, Object> properties)   // Find all upstream clusters reachable from the client Set<String> upstreamClusters(Map<String, Object> properties)   // Find the local offsets corresponding to the latest checkpoint from a specific upstream consumer group. Map<TopicPartition, OffsetAndMetadata> translateOffsets(Map<String, Object> properties,             String targetClusterAlias, String consumerGroupId, Duration timeout)

The utility class assumes DefaultReplicationPolicy is used for replication.

实用程序类假定DefaultReplicationPolicy用于复制。

MirrorClient

The RemoteClusterUtils class wraps a low-level MirrorClient, with the following exposed methods:

RemoteClusterUtils类包装了具有以下公开方法的低级MirrorClient：

MirrorClient(Map<String, Object> props) ...   void close() ...   ReplicationPolicy replicationPolicy() ...   int replicationHops(String upstreamClusterAlias) ...   Set<String> heartbeatTopics() ...   Set<String> checkpointTopics() ...   // Finds upstream clusters, which may be multiple hops away, based on incoming heartbeats. Set<String> upstreamClusters() ...   // Find all remote topics on the cluster Set<String> remoteTopics() ...   // Find all remote topics from the given source Set<String> remoteTopics(String source) ...   // Find the local offsets corresponding to the latest checkpoint from a specific upstream consumer group. Map<TopicPartition, OffsetAndMetadata> remoteConsumerOffsets(String consumerGroupId,             String remoteClusterAlias, Duration timeout) ...

Replication Policies and Filters（复制策略和过滤器）

A ReplicationPolicy defines what a "remote topic" is and how to interpret it. This should generally be consistent across an organization. The interface is as follows:

一个ReplicationPolicy  定义什么是“远程主题”，以及如何解释它。这在整个组织中通常应该是一致的。接口如下：

/** Defines which topics are "remote topics", e.g. "us-west.topic1". */ public interface ReplicationPolicy {       /** How to rename remote topics; generally should be like us-west.topic1. */     String formatRemoteTopic(String sourceClusterAlias, String topic);       /** Source cluster alias of given remote topic, e.g. "us-west" for "us-west.topic1".         Returns null if not a remote topic.     */     String topicSource(String topic);       /** Name of topic on the source cluster, e.g. "topic1" for "us-west.topic1".         Topics may be replicated multiple hops, so the immediately upstream topic         may itself be a remote topic.         Returns null if not a remote topic.     */     String upstreamTopic(String topic);       /** The name of the original source-topic, which may have been replicated multiple hops.         Returns the topic if it is not a remote topic.     */     String originalTopic(String topic);       /** Internal topics are never replicated. */     boolean isInternalTopic(String topic); }

The DefaultReplicationPolicy specifies the <source>.<topic> convention described throughout this document. LegacyReplicationPolicy causes MM2 to mimic legacy MirrorMaker behavior.

DefaultReplicationPolicy指定了本文档中描述的<source>.<topic>约定。LegacyReplicationPolicy使MM2模仿旧版MirrorMaker行为。

Additionally, several Filters control the behavior of MM2 at a high level, including which topics to replicate. The DefaultTopicFilter, DefaultGroupFilter, and DefaultConfigPropertyFilter classes honor whitelists and blacklists via regex patterns. 

此外，几个过滤器可以高层控制MM2的行为，包括要复制的主题。DefaultTopicFilter，DefaultGroupFilter和DefaultConfigPropertyFilter类通过正则表达式模式接受白名单和黑名单。 

/** Defines which topics should be replicated. */ public interface TopicFilter {       boolean shouldReplicateTopic(String topic); }   /** Defines which consumer groups should be replicated. */ public interface GroupFilter {       boolean shouldReplicateGroup(String group); }   /** Defines which topic configuration properties should be replicated. */ public interface ConfigPropertyFilter {       boolean shouldReplicateConfigProperty(String prop); }

 

Connectors（连接器）

Both source and sink connectors are provided to enable complex flows between multiple Kafka clusters and across data centers via existing Kafka Connect clusters. Generally, each data center has a single Connect cluster and a primary Kafka cluster, K.

提供了源连接器和接收器连接器，以通过现有的Kafka Connect群集实现多个Kafka群集之间以及跨数据中心的复杂流。通常，每个数据中心都有一个Connect集群和一个主要的 Kafka集群K。

MirrorSourceConnector workers replicate a set of topics from a single source cluster into the primary cluster.

MirrorSourceConnector workers将一组主题从单个源集群复制到主集群。

MirrorSinkConnector workers consume from the primary cluster and replicate topics to a single sink cluster.

MirrorSinkConnector workers从主群集使用并将主题复制到单个接收器群集。

MirrorCheckpointConnector emits consumer offset checkpoints.

MirrorCheckpointConnector 发出消费端偏移检查点。

MirrorHeartbeatConnector emits heartbeats.

MirrorHeartbeatConnector发出心跳。

The source and sink connectors contain a pair of producers and consumers to replicate records, and a pair of AdminClients to propagate configuration/ACL changes.

源连接器和接收器连接器包含用于复制记录的一对生产者和使用者，以及用于传播配置/ ACL更改的一对AdminClient。

 

MirrorSourceConnector and workers

MirrorSourceConnector和workers

By combining SourceConnectors and SinkConnectors, a single Connect cluster can manage replication across multiple Kafka clusters.

通过组合SourceConnectors和SinkConnectors，单个Connect群集可以管理跨多个Kafka群集的复制。

Connectors (C) replicate data between Kafka clusters.

连接器（C）在Kafka群集之间复制数据。

For cross-datacenter replication (XDCR), each datacenter should have a single Connect cluster which pulls records from the other data centers via source connectors. Replication may fan-out within each datacenter via sink connectors.

对于跨数据中心复制（XDCR），每个数据中心应具有一个Connect群集，该群集通过源连接器从其他数据中心提取记录。复制可能会通过接收器连接器散布在每个数据中心内。

 

cross-datacenter replication of multiple clusters

多个集群的跨数据中心复制

Connectors can be configured to replicate specific topics via a whitelist or regex. For example, clusters A1 and B1 above might contain only a subset of topics from K1, K2, or K3.

可以将连接器配置为通过白名单或正则表达式复制特定主题。例如，上面的群集A1和B1可能仅包含来自K1，K2或K3的主题的子集。

Connector Configuration Properties(连接器配置属性)

Properties common to the SourceConnector(s) and SinkConnector:

SourceConnector和SinkConnector共有的属性：

property(属性)	default value(默认值)	description(描述)
name(名称)	required(必须)	name of the connector, e.g. "us-west->us-east" 连接器的名称，例如“ us-west-> us-east”
topics(主题)	empty string(空字符串)	regex of topics to replicate, e.g. "topic1|topic2|topic3". Comma-separated lists are also supported. 要复制的主题的正则表达式，例如“ topic1 | topic2 | topic3”。还支持以逗号分隔的列表。
topics.blacklist(topic黑名单)	".*\.internal, .*\.replica, __consumer_offsets" or similar ".*\.internal, .*\.replica, __consumer_offsets"或类似内容	topics to exclude from replication 要排除的主题
groups(组)	empty strin(空字符串)	regex of groups to replicate, e.g. ".*" 要复制的组的正则表达式，例如“.*”
groups.blacklist(groups黑名单)	empty string(空字符串)	groups to exclude from replication 要排除的组
source.cluster.alias	required(必须)	name of the cluster being replicated 要被复制的集群别名
target.cluster.alias	required(必须)	name of the downstream Kafka cluster 下游Kafka集群别名
source.cluster.bootstrap.servers	required(必须)	upstream cluster to replicat 上游集群进行复制
target.cluster.bootstrap.servers	required(必须)	downstream cluster 下游集群
sync.topic.configs.enabled	true	whether or not to monitor source cluster for configuration changes 是否监视源集群的配置更改
sync.topic.acls.enabled	true	whether to monitor source cluster ACLs for change 是否监视源集群ACL的更改
emit.heartbeats.enabled	true	connector should periodically emit heartbeats 连接器应定期发出心跳
emit.heartbeats.interval.seconds	5 (seconds) 5秒	frequency of heartbeats 心跳频率
emit.checkpoints.enabled	true	connector should periodically emit consumer offset information 连接器应定期发出消费端偏移量信息
emit.checkpoints.interval.seconds	5 (seconds) 5秒	frequency of checkpoints 检查点的频率
refresh.topics.enabled	true	connector should periodically check for new topics 连接器应定期检查新主题
refresh.topics.interval.seconds	5 (seconds) 5秒	frequency to check source cluster for new topics 检查源群集中是否有新主题的频率
refresh.groups.enabled	true	connector should periodically check for new consumer groups 连接器应定期检查新的消费组
refresh.groups.interval.seconds	5 (seconds) 5秒	frequency to check source cluster for new consumer groups 检查源集群新的消费组频率
readahead.queue.capacity	500 (records) 500条	number of records to let consumer get ahead of producer 让消费者领先于生产者的记录数
replication.policy.class	org.apache.kafka.connect.mirror.DefaultReplicationPolicy	use LegacyReplicationPolicy to mimic legacy MirrorMake 使用LegacyReplicationPolicy模仿旧版MirrorMaker
heartbeats.topic.retention.ms	1 day	used when creating heartbeat topics for the first time 首次创建心跳主题时使用
checkpoints.topic.retention.ms	1 day	used when creating checkpoint topics for the first time 首次创建检查点主题时使用
offset.syncs.topic.retention.ms	max long	used when creating offset sync topic for the first time 首次创建偏移同步主题时使用
replication.factor	2	used when creating remote topics 在创建远程主题时使用

In addition, internal client properties can be overridden as follows:

此外，内部客户端属性可以按如下方式覆盖：

property	description
source.cluster.consumer.*	overrides for the source-cluster consumer 覆盖源集群消费者
source.cluster.producer.*	overrides for the source-cluster producer 覆盖源集群生产者
source.cluster.admin.*	overrides for the source-cluster admin 覆盖源集群管理员
target.cluster.consumer.*	overrides for the target-cluster consumer 覆盖目标集群消费者
target.cluster.producer.*	overrides for the target-cluster producer 覆盖目标集群生产者
target.cluster.admin.*	overrides for the target-cluster admin 覆盖目标集群管理员

 

These properties are specified in the MirrorConnectorConfig class, which is shared by all MirrorMaker connectors (MirrorSourceConnector, MirrorCheckpointConnector, etc). Some properties are relevant to specific Connectors, e.g. "emit.heartbeats" is only relevant to the MirrorHeartbeatConnector.

这些属性在MirrorConnectorConfig类中指定，该类被所有MirrorMaker连接器（MirrorSourceConnector，MirrorCheckpointConnector等）共享。一些属性与特定的连接器有关，例如，“ emit.heartbeats”仅与MirrorHeartbeatConnector有关。

Example Configuration（配置示例）

A sample configuration file ./config/connect-mirror-source.properties is provided:

提供了一个示例配置文件./config/connect-mirror-source.properties：

name = local-mirror-source topics = .* connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector tasks.max = 1     # for demo, source and target clusters are the same source.cluster.alias = upstream source.cluster.bootstrap.servers = localhost:9092 target.cluster.bootstrap.servers = localhost:9092     # use ByteArrayConverter to ensure that records are not re-encoded key.converter = org.apache.kafka.connect.converters.ByteArrayConverter value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

This enables running standalone MM2 connectors as follows:

这样可以运行独立的MM2连接器，如下所示：

$ ./bin/connect-standalone.sh ./config/connect-standalone.properties ./config/connect-mirror-source.properties

Legacy MirrorMaker Configuration（旧版MirrorMaker配置）

By default, MM2 enables several features and best-practices that are not supported by legacy MirrorMaker; however, these can all be turned off. The following configuration demonstrates running MM2 in "legacy mode":

默认情况下，MM2启用了旧版MirrorMaker不支持的多种功能和最佳实践。但是，这些都可以关闭。以下配置演示了以“旧版模式”运行MM2：

 

 

--%<----     # emulate legacy mirror-maker replication.policy.class = org.apache.kafka.mirror.LegacyReplicationPolicy     # disable all new features refresh.topics.enabled = false refresh.groups.enabled = false emit.checkpoints.enables = false emit.heartbeats.enabled = false sync.topic.configs.enabled = false sync.topic.acls.enabled = false     --%<----

MirrorMaker Clusters（MirrorMaker群集）

The MirrorMaker.java driver class and ./bin/connect-mirror-maker.sh script implement a distributed MM2 cluster which does not depend on an existing Connect cluster. Instead, MM2 cluster nodes manage Connect workers internally based on a high-level configuration file. The configuration file is needed to identify each Kafka cluster. A sample MirrorMakerConfig properties file will be provided in ./config/mirror-maker.properties:

MirrorMaker.java驱动程序类和./bin/connect-mirror-maker.sh脚本实现不依赖于现有Connect群集的分布式MM2群集。而是，MM2群集节点根据高级配置文件在内部管理Connect worker。需要配置文件来标识每个Kafka集群。./config/mirror-maker.properties中将提供一个样本MirrorMakerConfig属性文件：

clusters = primary, backup cluster.primary.bootstrap.servers = localhost:9091 cluster.backup.bootstrap.servers = localhost:9092

This can be run as follows:

可以如下运行：

$ ./bin/connect-mirror-maker.sh ./config/mirror-maker.properties

Internally, the MirrorMaker driver sets up MirrorSourceConnectors, MirrorCheckpointConnectors, etc between each pair of clusters, based on the provided configuration file. For example, the above configuration results in two MirrorSourceConnectors: one replicating from primary→backup and one from backup→primary. The configuration for the primary→backup connector is automatically populated as follows:

在内部，MirrorMaker驱动程序根据提供的配置文件在每对集群之间设置MirrorSourceConnectors，MirrorCheckpointConnectors等。例如，以上配置导致两个MirrorSourceConnector：一个从primary→backup复制，一个从backup→primary复制。将自动填充primary→backup连接器的配置，如下所示：

 

name = MirrorSourceConnector connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector source.cluster.alias = primary target.cluster.alias = backup source.cluster.bootstrap.servers = localhost:9091 target.cluster.bootstrap.servers = localhost:9092 key.converter.class = org.apache.kafka.connect.converters.ByteArrayConverter value.converter.class = org.apache.kafka.connect.converters.ByteArrayConverter

The MirrorMaker properties file can specify static configuration properties for each connector:

MirrorMaker属性文件可以为每个连接器指定静态配置属性：

clusters = primary, backup primary.bootstrap.servers = localhost:9091 backup.bootstrap.servers = localhost:9092 primary->backup.topics = .* primary->backup.emit.heartbeats.enabled = false

In this case, two MirrorSourceConnectors, two MirrorCheckpointConnectors, and two MirrorHeartbeatConnectors will be created by the driver and the primary→backup connectors will start replicating all source topics at launch, with heartbeats disabled. Sub-configurations like primary->backup.x.y.z will be applied to all connectors in that source→target flow (MirrorSourceConnector, MirrorCheckpointConnector), avoiding the need to configure each connector individually.

在这种情况下，驱动程序将创建两个MirrorSourceConnectors，两个MirrorCheckpointConnectors和两个MirrorHeartbeatConnectors，并且primary→backup连接器将在启动时开始复制所有源主题，并且禁用心跳。诸如primary-> backup.xyz之类的子配置将应用于该源→目标流（MirrorSourceConnector，MirrorCheckpointConnector）中的所有连接器，而无需分别配置每个连接器。

The MirrorMaker driver class is similar to the existing ConnectDistributed driver, except it runs multiple DistributedHerders, one for each Kafka cluster. The herders coordinate via their respective Kafka clusters – otherwise they are completely independent from each other (i.e. are separate logical Connect clusters). This driver makes it possible to bring up a "MirrorMaker cluster" with replication flows between multiple Kafka clusters without configuring each Connector, Worker, and Herder manually.

MirrorMaker驱动程序类与现有的ConnectDistributed驱动程序类似，不同之处在于它运行多个DistributedHerders，每个Kafka群集一个。Herders通过各自的Kafka集群进行协调-否则它们彼此完全独立（即，是单独的逻辑Connect集群）。使用该驱动程序，可以在多个Kafka群集之间启动具有复制流的“ MirrorMaker群集”，而无需手动配置每个Connector，Worker和Herder。

MirrorMaker Configuration Properties(MirrorMaker配置属性)

The high-level configuration file required by the MirrorMaker driver supports the following properties:

MirrorMaker驱动程序所需的高级配置文件支持以下属性：

property	default value	description
clusters	required(必须)	comma-separated list of Kafka cluster "aliases" 以逗号分隔的Kafka群集“别名”列表
cluster.bootstrap.servers	required(必须)	connection information for the specific cluster 特定集群的连接信息
cluster.x.y.z	n/a	passed to workers for a specific cluster 传递给特定集群的工人
source->target.x.y.z	n/a	passed to a specific connector 传递到特定的连接器

 

Walkthrough: Running MirrorMaker 2.0（演练：运行MirrorMaker 2.0）

There are four ways to run MM2:

有四种运行MM2的方法：

• As a dedicated MirrorMaker cluster.
• 作为专用的MirrorMaker群集。
• As a Connector in a distributed Connect cluster.
• 作为分布式Connect群集中的连接器。
• As a standalone Connect worker.
• 作为独立的Connect工作者。
• In legacy mode using existing MirrorMaker scripts.
• 在旧模式下，使用现有的MirrorMaker脚本。

Running a dedicated MirrorMaker cluster（运行专用的MirrorMaker集群）

In this mode, MirrorMaker does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers.

在这种模式下，MirrorMaker不需要现有的Connect群集。而是由高级驱动程序管理一组Connect worker。

First, specify Kafka cluster information in a configuration file:

首先，在配置文件中指定Kafka集群信息：

# mm2.properties clusters = us-west, us-east us-west.bootstrap.servers = host1:9092 us-east.bootstrap.servers = host2:9092

Optionally, you can override default MirrorMaker properties:

（可选）您可以覆盖默认的MirrorMaker属性：

topics = .* groups = .* emit.checkpoints.interval.seconds = 10

You can also override default properties for specific clusters or connectors:

您还可以覆盖特定群集或连接器的默认属性：

us-west.offset.storage.topic = mm2-offsets us-west->us-east.emit.heartbeats.enabled = false

Second, launch one or more MirrorMaker cluster nodes:

其次，启动一个或多个MirrorMaker群集节点：

$ ./bin/connect-mirror-maker.sh mm2.properties

Running a standalone MirrorMaker connector（运行独立的MirrorMaker连接器）

In this mode, a single Connect worker runs MirrorSourceConnector. This does not support multinode clusters, but is useful for small workloads or for testing.

在此模式下，单个Connect工作者运行MirrorSourceConnector。这不支持多节点群集，但是对于小型工作负载或测试很有用。

First, create a "worker" configuration file:

首先，创建一个“工作者”配置文件：

# worker.properties bootstrap.servers = host2:9092

An example is provided in ./config/connect-standalone.properties.

./config/connect-standalone.properties中提供了一个示例。

Second, create a "connector" configuration file:

其次，创建一个“连接器”配置文件：

# connector.properties name = local-mirror-source connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector tasks.max = 1 topics=.*   source.cluster.alias = upstream source.cluster.bootstrap.servers = host1:9092 target.cluster.bootstrap.servers = host2:9092   # use ByteArrayConverter to ensure that in and out records are exactly the same key.converter = org.apache.kafka.connect.converters.ByteArrayConverter value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

Finally, launch a single Connect worker:

最后，启动一个Connect工作者：

$ ./bin/connecti-standalone.sh worker.properties connector.properties

Running MirrorMaker in a Connect cluster（在Connect群集中运行MirrorMaker）

If you already have a Connect cluster, you can configure it to run MirrorMaker connectors.

如果您已经有一个Connect群集，则可以对其进行配置以运行MirrorMaker连接器。

There are four such connectors:

有四个这样的连接器：

• MirrorSourceConnector
• MirrorSinkConnector (coming soon)
• MirrorCheckpointConnector
• MirrorHeartbeatConnector

Configure these using the Connect REST API:

使用Connect REST API进行配置：

PUT /connectors/us-west-source/config HTTP/1.1   {     "name": "us-west-source",     "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",     "source.cluster.alias": "us-west",     "target.cluster.alias": "us-east",     "source.cluster.bootstrap.servers": "us-west-host1:9091",     "topics": ".*" }

Running MirrorMaker in legacy mode（在旧版模式下运行MirrorMaker）

After legacy MirrorMaker is deprecated, the existing ./bin/kafka-mirror-maker.sh scripts will be updated to run MM2 in legacy mode:

不推荐使用旧版MirrorMaker之后，将更新现有的./bin/kafka-mirror-maker.sh脚本以在旧版模式下运行MM2：

$ ./bin/kafka-mirror-maker.sh --consumer consumer.properties --producer producer.properties

 

Compatibility, Deprecation, and Migration Plan（兼容性，弃用和迁移计划）

A new version of ./bin/kafka-mirror-maker.sh will be implemented to run MM2 in "legacy mode", i.e. with new features disabled and supporting existing options and configuration properties. Existing deployments will be able to upgrade to MM2 seamlessly.

将实施新版本的./bin/kafka-mirror-maker.sh以在“旧版模式”下运行MM2，即禁用新功能并支持现有选项和配置属性。现有部署将能够无缝升级到MM2。

The existing MirrorMaker source will be removed from Kafka core project. MM2 will be added to the connect project under a new module "mirror" and package "org.apache.kafka.connect.mirror".

现有的MirrorMaker源文件将从Kafka核心项目中删除。MM2将在新模块“ mirror”和包“ org.apache.kafka.connect.mirror”下添加到connect项目。

Deprecation will occur in three phases:

弃用将分为三个阶段：

• Phase 1 (targeting next Apache Kafka release): All MirrorMaker 2.0 Java code is added to ./connect/mirror/.
• 阶段1（针对Apache Kafka的下一个发行版）：所有MirrorMaker 2.0 Java代码都已添加到./connect/mirror/。
• Phase 2 (subsequent release): Legacy MirrorMaker Scala code is deprecated, but kept in place. Sample MM2 scripts and configuration files are added to ./bin/ and ./config/.
• 第2阶段（后续版本）：不推荐使用旧版MirrorMaker Scala代码，但仍保留在原位。样本MM2脚本和配置文件已添加到./bin/和./config/中。
• Phase 3 (subsequent release): Legacy MirrorMaker Scala code is removed from Apache Kafka. A new ./bin/kafka-mirror-maker.sh script is provided which replaces and emulates the legacy script.
• 阶段3（后续版本）：从Apache Kafka中删除了旧版MirrorMaker Scala代码。提供了一个新的./bin/kafka-mirror-maker.sh脚本，该脚本替代并模仿了旧脚本。

Future Work（未来的工作）

• Command-line tools wrapping RemoteClusterUtils. Document cross-cluster migration procedure.
• 包装RemoteClusterUtils的命令行工具。文档跨集群迁移过程。
• Broker and KafkaConsumer support for unambiguous remote topic names, e.g. with a reserved separator character.
• Broker和KafkaConsumer支持明确的远程主题名称，例如带有保留的分隔符。
• CheckpointSinkConnector/Task to support upstream checkpointing when a single Connect cluster is used.
• CheckpointSinkConnector /任务，用于在使用单个Connect群集时支持上游检查点。
• Additional metrics.
• 其他指标。
• Support reconfiguration of running Tasks without stop, start.
• 支持重新配置正在运行的任务，而无需停止，启动。
• Exactly-once semantics in Connect and MM2.
• Connect和MM2中的精确一次性语义。
• Prevent duplicate streams from "diamond problem" (fan-out-fan-in).
• 防止来自“钻石问题”（扇出扇入）的重复流。
• Support multiple clusters/workers/herders in ConnectDistributed to obviate MM2-specific driver.
• 在ConnectDistributed中支持多个群集/workers/herders，以消除特定于MM2的驱动程序。

Rejected Alternatives（拒绝的选择）

• We could release this as an independent project, but we feel that cluster replication should be one of Kafka's fundamental features.
• 我们可以将其作为一个独立的项目发布，但是我们认为集群复制应该是Kafka的基本功能之一。
• We could deprecate MirrorMaker but keep it around. However, we see this as a natural evolution of MirrorMaker, not an alternative solution.
• 我们可以弃用MirrorMaker，但可以保留它。但是，我们认为这是MirrorMaker的自然演变，而不是替代解决方案。
• We could update MirrorMaker rather than completely rewrite it. However, we'd end up recreating many features provided by the Connect framework, including the REST API, configuration, metrics, worker coordination, etc.
• 我们可以更新MirrorMaker，而不是完全重写它。但是，我们最终将重新创建Connect框架提供的许多功能，包括REST API，配置，指标，工作人员协调等。
• We could build on Uber's uReplicator, which solves some of the problems with MirrorMaker. However, uReplicator uses Apache Helix to provide features that overlap with Connect, e.g. REST API, live configuration changes, cluster management, coordination etc. A native MirrorMaker should only use native parts of Apache Kafka. That said, uReplicator is a major inspiration for the MM2 design.
• 我们可以基于Uber的uReplicator来解决MirrorMaker的一些问题。但是，uReplicator使用Apache Helix提供与Connect重叠的功能，例如REST API，实时配置更改，集群管理，协调等。本机MirrorMaker应该仅使用Apache Kafka的本机部分。也就是说，uReplicator是MM2设计的主要灵感来源。
• We could provide a high-level REST API for remote-controlling a MirrorMaker cluster. However, this has overlapping concerns with Connect.
• 我们可以提供用于远程控制MirrorMaker集群的高级REST API。但是，这与Connect有重叠的问题。
• Instead of separate connectors for heartbeats and checkpoints, we could do everything in a single connector. However, this restricts how Converters and Transformations can be used. For example, you might want a replication pipeline that converts to JSON, but that doesn't necessarily mean you also want heartbeats to be JSON-encoded. It's possible to create a separate KafkaProducer within MirrorSourceConnector to emit Heartbeats and Checkpoints without going through the Connect transformation pipeline, but I don't want to introduce the additional configuration properties for a whole new producer.
• 代替用于心跳和检查点的单独连接器，我们可以在单个连接器中完成所有操作。但是，这限制了转换器和转换的使用方式。例如，您可能需要复制管道转换为JSON，但这并不一定意味着您还希望对心跳进行JSON编码。可以在MirrorSourceConnector中创建一个单独的KafkaProducer来发出心跳和检查点，而无需通过Connect转换管道，但是我不想为整个新生产者引入其他配置属性。