转载-zookeeper在kafka中的作用

1)Broker注册

  Broker在zookeeper中保存为一个临时节点,节点的路径是/brokers/ids/[brokerid],每个节点会保存对应broker的IP以及端口等信息.

  2)Topic注册

  在kafka中,一个topic会被分成多个区并被分到多个broker上,分区的信息以及broker的分布情况都保存在zookeeper中,根节点路径为/brokers/topics,每个topic都会在topics下建立独立的子节点,每个topic节点下都会包含分区以及broker的对应信息,例如下图中的状态

  3)生产者负载均衡

  当Broker启动时,会注册该Broker的信息,以及可订阅的topic信息。生产者通过注册在Broker以及Topic上的watcher动态的感知Broker以及Topic的分区情况,从而将Topic的分区动态的分配到broker上.

  4)消费者

  kafka有消费者分组的概念,每个分组中可以包含多个消费者,每条消息只会发给分组中的一个消费者,且每个分组之间是相互独立互不影响的。

  5)消费者与分区的对应关系

  对于每个消费者分组,kafka都会为其分配一个全局唯一的Group ID,分组内的所有消费者会共享该ID,kafka还会为每个消费者分配一个consumer ID,通常采用hostname:uuid的形式。在kafka的设计中规定,对于topic的每个分区,最多只能被一个消费者进行消费,也就是消费者与分区的关系是一对多的关系。消费者与分区的关系也被存储在zookeeper中节点的路劲为 /consumers/[group_id]/owners/[topic]/[broker_id-partition_id],该节点的内容就是消费者的Consumer ID

  6)消费者负载均衡

  消费者服务启动时,会创建一个属于消费者节点的临时节点,节点的路径为 /consumers/[group_id]/ids/[consumer_id],该节点的内容是该消费者订阅的Topic信息。每个消费者会对/consumers/[group_id]/ids节点注册Watcher监听器,一旦消费者的数量增加或减少就会触发消费者的负载均衡。消费者还会对/brokers/ids/[brokerid]节点进行监听,如果发现服务器的Broker服务器列表发生变化,也会进行消费者的负载均衡

  7)消费者的offset

  在kafka的消费者API分为两种(1)High Level Api:由zookeeper维护消费者的offset (2) Low Level API,自己的代码实现对offset的维护。由于自己维护offset往往比较复杂,所以多数情况下都是使用High Level的APIoffset在zookeeper中的节点路径为/consumers/[group_id]/offsets/[topic]/[broker_id-part_id],该节点的值就是对应的offset

  二、kakfa在zookeeper中的结构

  kafka.javaapi.consumer.ZookeeperConsumerConnector.java

  /**

  * This class handles the consumers interaction with zookeeper

  *

  * Directories:

  * 1. Consumer id registry:

  * /consumers/[group_id]/ids[consumer_id] -> topic1,...topicN

  * A consumer has a unique consumer id within a consumer group. A consumer registers its id as an ephemeral znode

  * and puts all topics that it subscribes to as the value of the znode. The znode is deleted when the client is gone.

  * A consumer subscribes to event changes of the consumer id registry within its group.

  *

  * The consumer id is picked up from configuration, instead of the sequential id assigned by ZK. Generated sequential

  * ids are hard to recover during temporary connection loss to ZK, since it's difficult for the client to figure out

  * whether the creation of a sequential znode has succeeded or not. More details can be found at

  *

  * 2. Broker node registry:

  * /brokers/[0...N] --> { "host" : "host:port",

  * "topics" : {"topic1": ["partition1" ... "partitionN"], ...,

  * "topicN": ["partition1" ... "partitionN"] } }

  * This is a list of all present broker brokers. A unique logical node id is configured on each broker node. A broker

  * node registers itself on start-up and creates a znode with the logical node id under /brokers. The value of the znode

  * is a JSON String that contains (1) the host name and the port the broker is listening to, (2) a list of topics that

  * the broker serves, (3) a list of logical partitions assigned to each topic on the broker.

  * A consumer subscribes to event changes of the broker node registry.

  *

  * 3. Partition owner registry:

  * /consumers/[group_id]/owner/[topic]/[broker_id-partition_id] --> consumer_node_id

  * This stores the mapping before broker partitions and consumers. Each partition is owned by a unique consumer

  * within a consumer group. The mapping is reestablished after each rebalancing.

  *

  * 4. Consumer offset tracking:

  * /consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] --> offset_counter_value

  * Each consumer tracks the offset of the latest message consumed for each partition.

  *

  */

阅读更多
换一批

没有更多推荐了,返回首页