Broker node registration、Consumer registration algorithm、Consumer rebalancing algorithm

原创 2013年12月02日 16:00:26

Broker node registration

The broker nodes are basically independent, so they only publish information about what they have. When a broker joins, it registers itself under the broker node registry directory and writes information about its host name and port. The broker also register the list of existing topics and their logical partitions in the broker topic registry. New topics are registered dynamically when they are created on the broker.

Consumer registration algorithm

When a consumer starts, it does the following:

  1. Register itself in the consumer id registry under its group.
  2. Register a watch on changes (new consumers joining or any existing consumers leaving) under the consumer id registry. (Each change triggers rebalancing among all consumers within the group to which the changed consumer belongs.)
  3. Register a watch on changes (new brokers joining or any existing brokers leaving) under the broker id registry. (Each change triggers rebalancing among all consumers in all consumer groups.)
  4. If the consumer creates a message stream using a topic filter, it also registers a watch on changes (new topics being added) under the broker topic registry. (Each change will trigger re-evaluation of the available topics to determine which topics are allowed by the topic filter. A new allowed topic will trigger rebalancing among all consumers within the consumer group.)
  5. Force itself to rebalance within in its consumer group.

Consumer rebalancing algorithm

The consumer rebalancing algorithms allows all the consumers in a group to come into consensus on which consumer is consuming which partitions. Consumer rebalancing is triggered on each addition or removal of both broker nodes and other consumers within the same group. For a given topic and a given consumer group, broker partitions are divided evenly among consumers within the group. A partition is always consumed by a single consumer. This design simplifies the implementation. Had we allowed a partition to be concurrently consumed by multiple consumers, there would be contention on the partition and some kind of locking would be required. If there are more consumers than partitions, some consumers won't get any data at all. During rebalancing, we try to assign partitions to consumers in such a way that reduces the number of broker nodes each consumer has to connect to.

Each consumer does the following during rebalancing:

   1. For each topic T that Ci subscribes to 
   2.   let PT be all partitions producing topic T
   3.   let CG be all consumers in the same group as Ci that consume topic T
   4.   sort PT (so partitions on the same broker are clustered together)
   5.   sort CG
   6.   let i be the index position of Ci in CG and let N = size(PT)/size(CG)
   7.   assign partitions from i*N to (i+1)*N - 1 to consumer Ci
   8.   remove current entries owned by Ci from the partition owner registry
   9.   add newly assigned partitions to the partition owner registry
        (we may need to re-try this until the original partition owner releases its ownership)

When rebalancing is triggered at one consumer, rebalancing should be triggered in other consumers within the same group about the same time.

相关文章推荐

TNS-01185: Registration attempted from a remote node

状况:在节点一可以监听到ASM实例和数据库实例(racdb),但是节点二就只能监听到ASM实例,监听不到节点二的实例 监听日志: 这里和 local_listener参数有关,...

Oracle 11.2.0.4/12C新特性Valid Node Checking For Registration (VNCR)

防止非法数据库服务注册到重要监听

kafka消息服务的producer、broker、consumer的配置

server.properties配置: server.properties中所有配置参数说明(解释)如下列表: 参数 说明(解释) broker.id =0 每一个...
  • QH_JAVA
  • QH_JAVA
  • 2017年03月29日 11:40
  • 589

Kafka学习总结(四)——broker、consumer、producer主要配置

1、Broker配置 2、Consumer主要配置 /指定zookeeper的连接的字符串/ props.put(“zookeeper.connect”,”130.60.23.193:2181,...

【Leetcode Algorithm】Remove Nth Node From End of List

题目: Given a linked list, remove the nth node from the end of list and return its head. For...

Ambari学习10_ambari安装过程Registration with the server failed和[Errno 256] No more mirrors to try

今天重新安装ambari过程中,遇到了几个问题,耗费了我很长时间,在此记录一下 ambari重新安装可参考我之前的一篇随笔 http://www.cnblogs.com/6tian/p/40974...

Manual Installation and Registration of Java Plugin for Linux

Manual Installation and Registration of Java Plugin for Linux JDK DocumentationWhen you install the ...

C - Registration system

C - Registration system Time Limit:5000MS     Memory Limit:65536KB     64bit IO Format:%I64d & %I64...

code forces 4C Registration system (map 的简单用法)

C. Registration system time limit per test 5 seconds memory limit per test 64 megabytes ...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:Broker node registration、Consumer registration algorithm、Consumer rebalancing algorithm
举报原因:
原因补充:

(最多只允许输入30个字)