rocketMq 集群消费和广播消费

前言

rocketMq消费模式分为集群消费clustering和广播消费broadcast, 默认是clustering

消费方式:

• PullConsumer，由用户主动调用pull方法来获取消息，没有则返回
• PushConsumer，在启动后，Consumer客户端会主动循环发送Pull请求到broker，如果没有消息，broker会把请求放入等待队列，新消息到达后返回response。

消费模式:

• broadcast 消息会发送给group内所有consumer。
• clustering 每条消息只会发送给group内的一个consumer，集群模式的支持消费失败重发,消息队列数量与消费者关系：1个消费者可以消费多个队列，但1个消息队列只会被一个消费者消费；如果消费者数量大于消息队列数量，则有的消费者会消费不到消息

一、消费类图

二、集群消费

2.1 DefaultMQPushConsumer

先看下四个构造函数

   /**
     * Default constructor.
     */
    public DefaultMQPushConsumer() {
   
        this(MixAll.DEFAULT_CONSUMER_GROUP, null, new AllocateMessageQueueAveragely());
    }

    /**
     * Constructor specifying consumer group, RPC hook and message queue allocating algorithm.构造函数指定使用者组、RPC钩子和消息队列分配算法。
     *
     * @param consumerGroup Consume queue.
     * @param rpcHook RPC hook to execute before each remoting command.
     * @param allocateMessageQueueStrategy message queue allocating algorithm.
     */
    public DefaultMQPushConsumer(final String consumerGroup, RPCHook rpcHook,
        AllocateMessageQueueStrategy allocateMessageQueueStrategy) {
   
        this.consumerGroup = consumerGroup;
        this.allocateMessageQueueStrategy = allocateMessageQueueStrategy;
        defaultMQPushConsumerImpl = new DefaultMQPushConsumerImpl(this, rpcHook);
    }

    /**
     * Constructor specifying RPC hook.
     *
     * @param rpcHook RPC hook to execute before each remoting command.
     */
    public DefaultMQPushConsumer(RPCHook rpcHook) {
   
        this(MixAll.DEFAULT_CONSUMER_GROUP, rpcHook, new AllocateMessageQueueAveragely());
    }

    /**
     * Constructor specifying consumer group.
     *
     * @param consumerGroup Consumer group.
     */
    public DefaultMQPushConsumer(final String consumerGroup) {
   
//        这里采用平均散列队列算法
        this(consumerGroup, null, new AllocateMessageQueueAveragely());
    }

默认使用AllocateMessageQueueAveragely分配queue策略支持六种策略:

• AllocateMessageQueueConsistentHash 一致性哈希队列算法
• AllocateMessageQueueByMachineRoom 机房哈希队列算法
• AllocateMessageQueueByConfig 按配置分配消息队列
• AllocateMessageQueueAveragelyByCircle 循环平均哈希队列算法
• AllocateMessageQueueAveragely 平均散列队列算法
• AllocateMachineRoomNearby 机房近端优先级的分配策略代理

DefaultMQPushConsumer的start方法

@Override
    public void start() throws MQClientException {
   
//        服务启动=》
        this.defaultMQPullConsumerImpl.start();
    }

DefaultMQPullConsumerImpl的start方法

      public synchronized void start() throws MQClientException {
   
        switch (this.serviceState) {
   
            case CREATE_JUST:
                this.serviceState = ServiceState.START_FAILED;

                //1、基本的参数检查，group name不能是DEFAULT_CONSUMER
                this.checkConfig();

                //2、将DefaultMQPushConsumer的订阅信息copy到RebalanceService中
                this.copySubscription();
                //3、如果是cluster模式，修改InstanceName参数值为PID
                if (this.defaultMQPullConsumer.getMessageModel() == MessageModel.CLUSTERING) {
   
                    this.defaultMQPullConsumer.changeInstanceNameToPID();
                }

//              //4、新建一个MQClientInstance,客户端管理类，所有的i/o类操作由它管理
//              //缓存客户端和topic信息，各种service
                this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPullConsumer, this.rpcHook);

                this.rebalanceImpl.setConsumerGroup(this.defaultMQPullConsumer.getConsumerGroup());
                this.rebalanceImpl.setMessageModel(this.defaultMQPullConsumer.getMessageModel());
                //Queue分配策略
                //对于同一个group内的consumer，RebalanceImpl负责分配具体每个consumer应该消费哪些queue上的消息,以达到负载均衡的目的。
                // Rebalance支持多种分配策略，比如平均分配、一致性Hash等(具体参考AllocateMessageQueueStrategy实现类)。默认采用平均分配策略(AVG)。

                this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPullConsumer.getAllocateMessageQueueStrategy());
                this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
                //PullRequest封装实现类，封装了和broker的通信接口
                this.pullAPIWrapper = new PullAPIWrapper(
                    mQClientFactory,
                    this.defaultMQPullConsumer.getConsumerGroup(), isUnitMode());
//                注册过滤消息钩子方法  消息被客户端过滤时会回调hook
                this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);

                if (this.defaultMQPullConsumer.getOffsetStore() != null) {
   
                    this.offsetStore = this.defaultMQPullConsumer.getOffsetStore();
                } else {
   
                    switch (this.defaultMQPullConsumer.getMessageModel()) {
   
//                        广播消息本地持久化offse
                        case BROADCASTING:
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPullConsumer.getConsumerGroup());
                            break;
//                            集群模式持久化到broker
                        case CLUSTERING:
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPullConsumer.getConsumerGroup());
                            break;
                        default:
                            break;
                    }
                    this.defaultMQPullConsumer.setOffsetStore(this.offsetStore);
                }

//                offset加载=》如果是本地持久化会从文件中load
                this.offsetStore.load();

//                注册消费者=》
                boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPullConsumer.getConsumerGroup(), this);
                if (!registerOK) {
   
                    this.serviceState = ServiceState.CREATE_JUST;

                    throw new MQClientException("The consumer group[" + this.defaultMQPullConsumer.getConsumerGroup()
                        + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                        null);
                }

//              启动MQClientInstance，会启动PullMessageService和RebalanceService
                mQClientFactory.start();
                log.info("the consumer [{}] start OK", this.defaultMQPullConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PullConsumer service state not OK, maybe started once, "
                    + this.serviceState
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }

    }

MQClientInstance的start

   public void start() throws MQClientException {
   

        synchronized (this) {
   
            switch (this.serviceState) {
   
                case CREATE_JUST://仅创建不启动
                    this.serviceState = ServiceState.START_FAILED;
                    // If not specified,looking address from name server 如果启动的时候命令行没有指定name server的地址，就去获取
                    if (null == this.clientConfig.getNamesrvAddr()) {
   
//                        获取namesrv地址
                        this.mQClientAPIImpl.fetchNameServerAddr();
                    }
                    // Start request-response channel 启动请求响应的channel =》
                    this.mQClientAPIImpl.start();
                    // Start various schedule tasks 启动调度任务=》
                    this.startScheduledTask();
                    // Start pull service 启动pull消息服务=》
                    this.pullMessageService.start();
                    // Start rebalance service 启动负载均衡服务=》
                    this.rebalanceService.start();
                    // Start push service 启动push服务=》
                    this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
                    log.info("the client factory [{}] start OK", this.clientId);
                    this.serviceState = ServiceState.RUNNING;
                    break;
                case RUNNING:
                    break;
                case SHUTDOWN_ALREADY:
                    break;
                case START_FAILED:
                    throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
                default:
                    break;
            }
        }
    }

startScheduledTask() 开启几个定时任务

    private void startScheduledTask() {
   
        if (null == this.clientConfig.getNamesrvAddr()) {
   
            this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
   

                @Override
                public void run() {
   
                    try {
   
//                        更新namesrv地址
                        MQClientInstance.this.mQClientAPIImpl.fetchNameServerAddr()