4.rocketmq源代码学习----客户端消息消费(负载均衡）

rocketmq是通过队列来做负载均衡的
一个主题可以对应到多个队列，通过分配队列的数量来保障消费端的负载均衡，一个队列只会被分配给一台消费者，来保障消息不会被同一个消费者ID集群消费。

例如：TopicA有5个队列，消费者Group1部署了两台机器
则一台机器消费queue0、queue1、queue2
另外一台机器消费queue3、queue4
通过队列来保障了消息的负载均衡

代码如下：

负载均衡的关键代码为：ReblanceImpl.rebalanceByTopic()
整体思路为：
①从broker端获取 消费者+topic 的所有消费者id，这个消费者id就是启动时，注册的clientId：ip@port
②获取topic配置的所有队列信息，这个是从哪来的呢，就是MQClientInstance.startScheduledTask()中的updateTopicRouteInfoFromNameServer()，每隔10s从namesrv端获取主题的配置信息
③对消费者id、主题队列信息 按照升序排序
④根据第三步的两个数据做负载均衡，rocketmq提供的负载均衡策略有四种：

这几种策略无非是分配 当前消费者的所有id 消费 哪些队列
默认的策略为：AllocateMessageQueueAveragely

@Override
    public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
                                       List<String> cidAll) {
        if (currentCID == null || currentCID.length() < 1) {
            throw new IllegalArgumentException("currentCID is empty");
        }
        if (mqAll == null || mqAll.isEmpty()) {
            throw new IllegalArgumentException("mqAll is null or mqAll empty");
        }
        if (cidAll == null || cidAll.isEmpty()) {
            throw new IllegalArgumentException("cidAll is null or cidAll empty");
        }

        List<MessageQueue> result = new ArrayList<MessageQueue>();
        if (!cidAll.contains(currentCID)) {
            log.info("[BUG] ConsumerGroup: {} The consumerId: {} not in cidAll: {}", //
                    consumerGroup, //
                    currentCID,//
                    cidAll);
            return result;
        }

        int index = cidAll.indexOf(currentCID);//0
        int mod = mqAll.size() % cidAll.size();//6%4
        int averageSize =
                mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
                        + 1 : mqAll.size() / cidAll.size());
        int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
        int range = Math.min(averageSize, mqAll.size() - startIndex);
        for (int i = 0; i < range; i++) {
            result.add(mqAll.get((startIndex + i) % mqAll.size()));
        }
        return result;
    }

上面代码，调试下就知道大概是什么样的均衡策略，
举个例子，假设队列数为4个，当消费者数量分别为：2个、3个、4个时，分配如下：

Queue	2个消费者	3个消费者	4个消费者
queue0	consumer0	consumer0	consumer0
queue1	consumer0	consumer0	consumer1
queue2	consumer1	consumer1	consumer2
queue3	consumer1	consumer2	consumer3

⑤分配后，将调用ReblanceImpl.updateProcessQueueTableInRebalance()方法
该方法会和上一次的分配结果做对比，如果发生了变更，
a、如果是新增了队列，则新生成PullRequest，pullRequest有啥用，这就是拉取消息的关键代码
b、如果是减少了队列，会舍弃pullRequest，并设置dropped=true–》对应方法为：truncateMessageQueueNotMyTopic()

     private boolean updateProcessQueueTableInRebalance(final String topic, final Set<MessageQueue> mqSet) {
        boolean changed = false;
		//获取内存中上一次负载均衡的结果
        Iterator<Entry<MessageQueue, ProcessQueue>> it = this.processQueueTable.entrySet().iterator();
        //比较如果减少了队列，则移除，并设置dropped=true
        while (it.hasNext()) {
            Entry<MessageQueue, ProcessQueue> next = it.next();
            MessageQueue mq = next.getKey();
            ProcessQueue pq = next.getValue();

            if (mq.getTopic().equals(topic)) {
                if (!mqSet.contains(mq)) {
                    pq.setDropped(true);
                    if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                        it.remove();
                        changed = true;
                        log.info("doRebalance, {}, remove unnecessary mq, {}", consumerGroup, mq);
                    }
                }
                else if (pq.isPullExpired()) {
                    switch (this.consumeType()) {
                        case CONSUME_ACTIVELY:
                            break;
                        case CONSUME_PASSIVELY:
                            pq.setDropped(true);
                            if (this.removeUnnecessaryMessageQueue(mq, pq)) {
                                it.remove();
                                changed = true;
                                log.error(
                                        "[BUG]doRebalance, {}, remove unnecessary mq, {}, because pull is pause, so try to fixed it",
                                        consumerGroup, mq);
                            }
                            break;
                        default:
                            break;
                    }
                }
            }
        }

        List<PullRequest> pullRequestList = new ArrayList<PullRequest>();
        for (MessageQueue mq : mqSet) {
	        //比较如果新增了队列，则新生成pullRequest
            if (!this.processQueueTable.containsKey(mq)) {
                PullRequest pullRequest = new PullRequest();
                pullRequest.setConsumerGroup(consumerGroup);
                pullRequest.setMessageQueue(mq);
                pullRequest.setProcessQueue(new ProcessQueue());
				//获取当前应该从哪里开始消费
                long nextOffset = this.computePullFromWhere(mq);
                if (nextOffset >= 0) {
	                //设置pullRequest从哪里开始消费消息
                    pullRequest.setNextOffset(nextOffset);
                    pullRequestList.add(pullRequest);
                    changed = true;
                    this.processQueueTable.put(mq, pullRequest.getProcessQueue());
                    log.info("doRebalance, {}, add a new mq, {}", consumerGroup, mq);
                } else {
                    log.warn("doRebalance, {}, add new mq failed, {}", consumerGroup, mq);
                }
            }
        }
		//提交拉取消息请求
        this.dispatchPullRequest(pullRequestList);

        return changed;
    }
    
    @Override
    public void dispatchPullRequest(List<PullRequest> pullRequestList) {
        for (PullRequest pullRequest : pullRequestList) {
	       //提交立马拉取消息    
	        this.defaultMQPushConsumerImpl.executePullRequestImmediately(pullRequest);
            log.info("doRebalance, {}, add a new pull request {}", consumerGroup, pullRequest);
        }
    }

//DefaultMQPushConsumerImpl
 public void executePullRequestImmediately(final PullRequest pullRequest) {        
//调用PullMessageService提交pullRequest请求    
    this.mQClientFactory.getPullMessageService().executePullRequestImmediately(pullRequest);
    }

当新增了队列时，构造了PullRequest，从这儿调用了computePullFromWhere()获取消费进度()
之后调用PullMessageService提交了消费者请求