RocketMQ第二期：生产者、消费者启动原理

从代码层面解析生产者、消费者启动原理以及消息发送的原理。

源码调试

1、下载源码

RocketMQ源码
解压：unzip rocketmq-all-4.7.1-source-release.zip

2、设置环境变量

如果命令行使用的是zsh：

vim ~/.zshrc

添加下面两行配置：

export ROCKETMQ_HOME=你的rocketmq解压目录/distribution/target/rocketmq-4.7.0/rocketmq-4.7.0
#启动broker需要
export NAMESRV_ADDR=localhost:9876

source ~/.zshrc

3、创建topic

cd 你的rocketmq解压目录/distribution/target/rocketmq-4.7.0/rocketmq-4.7.0/bin
sh mqadmin updateTopic -n localhost:9876 -b localhost:10911 -t TopicTest

查看topic路由信息

cd 你的rocketmq解压目录/distribution/target/rocketmq-4.7.0/rocketmq-4.7.0/bin
sh mqadmin topicRoute -n localhost:9876 -t TopicTest

发送消息

/*
         * 实例化生产者并指定生产者组
         */
        DefaultMQProducer producer = new DefaultMQProducer("producer_group");
  
        /*
         *指定namesrv地址，不指定的话默认从系统环境变量NAMESRV_ADDR或者jvm参数rocketmq.namesrv.addr获取
         */
//        consumer.setNamesrvAddr("localhost:9876");
        //最大重试次数
//        producer.setRetryTimesWhenSendFailed(3);
        /*
         * Launch the instance.
         */
        producer.start();
  
        for (int i = 0; i < 10; i++) {
            try {
  
                /*
                 * Create a message instance, specifying topic, tag and message body.
                 */
                Message msg = new Message("TopicTest" /* Topic */,
                    "TagA" /* Tag */,
                    ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
                );
  
                /*
                 * Call send message to deliver message to one of brokers.
                 */
                SendResult sendResult = producer.send(msg);
                //单向发送，没有返回结果，不管成功与否
//                producer.sendOneway(msg);
                //异步+回调发送
//                producer.send(msg, new SendCallback() {
//
//                    @Override
//                    public void onSuccess(SendResult sendResult) {
//                        System.out.println("发送成功");
//                    }
//
//                    @Override
//                    public void onException(Throwable e) {
//                        System.out.println("发送异常");
//                    }
//                });
  
                System.out.printf("%s%n", sendResult);
            } catch (Exception e) {
                e.printStackTrace();
                Thread.sleep(1000);
            }
        }

消费消息

/*
         * 实例化消费者并指定消费者组，一个jvm进程同一个消费者组的消费者实例只能有一个
         */
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("consumer_group");
  
        /*
         *指定namesrv地址，不指定的话默认从系统环境变量NAMESRV_ADDR或者jvm参数rocketmq.namesrv.addr获取
         */
//        consumer.setNamesrvAddr("localhost:9876");
        //自定义instanceName，一个需要注意的地方：http://tiku-wiki.baijiahulian.com/pages/viewpage.action?pageId=13125888
//        consumer.setInstanceName("XUJIAN_MACBOOK");
        //设置消费模式：集群/广播
//        consumer.setMessageModel(MessageModel.BROADCASTING);
  
        /*
         * 设置从哪里开始消费
         */
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
        //设置最大重试次数
//        consumer.setMaxReconsumeTimes(3);
  
        /*
         * 设置订阅信息，‘*’代表所有的tag，多个tag用‘||’分隔
         */
        consumer.subscribe("TopicTest", "*");
  
        /*
         *  注册消息监听器，MessageListenerConcurrently（普通消息监听器）、MessageListenerOrderly（顺序消息监听器）
         */
        consumer.registerMessageListener(new MessageListenerConcurrently() {
  
            @Override
            /**
             * 注意到这里的msgs参数是个list，size>=1，可以批量消费，通过设置consumeMessageBatchMaxSize参数
             */
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
                ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                System.out.printf("msgBody: %s %n",new String(msgs.get(0).getBody()));
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
  
        /*
         *  Launch the consumer instance.
         */
        consumer.start();

consumer.subscribe(“TopicTest”, “*”);
多次调用可以订阅多个topic。

启动两个消费者：
idea设置如下，要不然不能启动两次一样的程序

生产者和消费者的实现都是用了“门面模式”。如
DefaultMQPushConsumerImpl是DefaultMQPushConsumer的一个门面。

源码解析

消费者启动原理

从consumer.start()说起。消息消费方式有push模式和pull模式，后面以push为例来进行讨论。

DefaultMQPushConsumerImpl#start

public synchronized void start() throws MQClientException {
        //判断服务状态
        switch (this.serviceState) {
            //刚刚创建
            case CREATE_JUST:
                log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
                    this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
                this.serviceState = ServiceState.START_FAILED;
 
                //1、检查消费者配置
                this.checkConfig();
 
                //把消费者订阅信息保存到内部的Map<topic,subscriptionData>中，和重试topic相关
                this.copySubscription();
 
                //如果是集群消费，用进程id替代instanceName属性
                if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
                    this.defaultMQPushConsumer.changeInstanceNameToPID();
                }
 
                /*
                 2、获取MQClientInstance对象，同一个clientId只能生成一个MQClientInstance。
                 */
                this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
 
                //设置消息消费负载均衡相关属性
                this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
                this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
                this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
                this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
 
                //创建消息拉取api wrapper
                this.pullAPIWrapper = new PullAPIWrapper(
                    mQClientFactory,
                    this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
                this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
 
                if (this.defaultMQPushConsumer.getOffsetStore() != null) {
                    this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
                } else {
                    switch (this.defaultMQPushConsumer.getMessageModel()) {
                        case BROADCASTING:
                            //广播消费，offset是保存在本地的
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        case CLUSTERING:
                            //集群消费，offset是保存在broker的
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        default:
                            break;
                    }
                    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
                }
                this.offsetStore.load();
 
                //3、根据不同的消息类型实例化不同的消费消息服务
                if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
                    this.consumeOrderly = true;
                    this.consumeMessageService =
                        new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
                } else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
                    this.consumeOrderly = false;
                    this.consumeMessageService =
                        new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
                }
                //4、启动消费消息服务，启动定时任务处理过期消息
                this.consumeMessageService.start();
 
                //5、注册消费者实例
                boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
                if (!registerOK) {
                    this.serviceState = ServiceState.CREATE_JUST;
                    this.consumeMessageService.shutdown();
                    throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
                        + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                        null);
                }
 
                //6、启动MQClientInstance实例
                mQClientFactory.start();
                log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
                    + this.serviceState
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }
 
        //7、如果订阅信息改变则需要更新本地缓存
        this.updateTopicSubscribeInfoWhenSubscriptionChanged();
        //8、在broker中检查client的配置信息
        this.mQClientFactory.checkClientInBroker();
        //9、发送心跳到所有的broker
        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
        //10、唤醒再均衡服务
        this.mQClientFactory.rebalanceImmediately();

MQClientInstance#getOrCreateMQClientInstance

/**
     * 获取或者创建MQClientInstance
     * @param clientConfig
     * @param rpcHook
     * @return
     */
    public MQClientInstance getOrCreateMQClientInstance(final ClientConfig clientConfig, RPCHook rpcHook) {
        //根据instanceName构造clientId
        String clientId = clientConfig.buildMQClientId();
        //根据clientId从map获取MQClientInstance
        MQClientInstance instance = this.factoryTable.get(clientId);
        if (null == instance) {
            instance =
                new MQClientInstance(clientConfig.cloneClientConfig(),
                    this.factoryIndexGenerator.getAndIncrement(), clientId, rpcHook);
            //如果之前存在clientId对应的MQClientInstance，则后面以该clientId启动的MQClientInstance不会被注册
            MQClientInstance prev = this.factoryTable.putIfAbsent(clientId, instance);
            if (prev != null) {
                instance = prev;
                log.warn("Returned Previous MQClientInstance for clientId:[{}]", clientId);
            } else {
                log.info("Created new MQClientInstance for clientId:[{}]", clientId);
            }
        }
 
        return instance;
    }

可以仔细看一下上面的代码，这里抛出两个问题。

问题1：putIfAbsent方法的作用是什么？
如果map中不存在key才会put，如果存在则返回该key对应的value。

问题2：首先判断instance为null的时候，new一个MQClientInstance，然后用putIfAbsent把它放到map里，之后为什么还要再判断putIfAbsent的返回值是不是空？
防止其他线程同时进入该方法（该方法并不是一个同步方法），但是先执行了this.factoryTable.putIfAbsent(clientId, instance);这个操作，这个问题主要是在生产者启动的时候会出现，而消费者启动的方法加了同步锁。

MQClientInstance#registerConsumer

public boolean registerConsumer(final String group, final MQConsumerInner consumer) {
        if (null == group || null == consumer) {
            return false;
        }
        //将消费者组和消费者实例的对应关系保存到map
        MQConsumerInner prev = this.consumerTable.putIfAbsent(group, consumer);
        if (prev != null) {
            //如果之前已经存在同样消费者组的消费者实例则注册失败
            log.warn("the consumer group[" + group + "] exist already.");
            return false;
        }
 
        return true;
    }

同一个JVM同一个消费者组只能有一个消费者实例

MQClientInstance#start

public void start() throws MQClientException {
 
        synchronized (this) {
            switch (this.serviceState) {
                case CREATE_JUST:
                    this.serviceState = ServiceState.START_FAILED;
                    // If not specified,looking address from name server
                    if (null == this.clientConfig.getNamesrvAddr()) {
                        this.mQClientAPIImpl.fetchNameServerAddr();
                    }
                    // Start request-response channel，启动mQClient网络请求响应相关的接口，依赖netty
                    this.mQClientAPIImpl.start();
                    // Start various schedule tasks，启动定时任务，心跳等
                    this.startScheduledTask();
                    // Start pull service，启动拉取消息服务
                    this.pullMessageService.start();
                    // Start rebalance service，启动消息负载均衡服务
                    this.rebalanceService.start();
                    // Start push service，启动消息发送者，为什么启动消费者的时候会启动producer？
                    this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
                    log.info("the client factory [{}] start OK", this.clientId);
                    this.serviceState = ServiceState.RUNNING;
                    break;
                case START_FAILED:
                    throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
                default:
                    break;
            }
        }
    }

疑问：为什么启动消费者的时候会启动producer？

解析：这个producer是内部的生产者，主要用于，广播模式下消费消息失败，将失败的消息重新投递到broker，从而达到重新消费失败消息的目的。

这里还有一个需要注意的地方，就是对consumer设置其instanceName属性出现的问题。见设置了instanceName属性，结果消费紊乱

总结

生产者启动流程：

1. 检查消费者配置
2. 获取MQClientInstance
3. 根据不同的消息类型实例化不同的消费消息服务
4. 启动消费消息服务，启动定时任务处理过期消息
5. 注册消费者实例
6. 启动MQClientInstance实例
7. 如果订阅信息改变则需要更新本地缓存
8. 在broker中检查client的配置信息
9. 发送心跳到所有的broker 唤醒再均衡服务
   

生产者启动原理

DefaultMQProducerImpl#start

public void start(final boolean startFactory) throws MQClientException {
        switch (this.serviceState) {
            case CREATE_JUST:
                this.serviceState = ServiceState.START_FAILED;
 
                //1、主要检查producerGroup
                this.checkConfig();
 
                if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
                    //设置instanceName
                    this.defaultMQProducer.changeInstanceNameToPID();
                }
 
                //2、获取MQClientInstance
                this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook);
 
                //3、注册producer
                boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
                if (!registerOK) {
                    this.serviceState = ServiceState.CREATE_JUST;
                    throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
                        + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                        null);
                }
 
                this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
 
                if (startFactory) {
                    /*
                    注意这里，DefaultMQProducerImpl的start方法有个startFactory参数，
                    由于MQClientInstance的start方法里又会调用到DefaultMQProducerImpl的start方法
                    不过传的startFactory参数是false，否则会造成循环调用。
                     */
                    mQClientFactory.start();
                }
 
                log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
                    this.defaultMQProducer.isSendMessageWithVIPChannel());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The producer service state not OK, maybe started once, "
                    + this.serviceState
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }
 
        //4、同步的方式向所有broker发送心跳
        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
 
        //5、启动定时任务处理DefaultMQProducer#request 发送request message（测试发送延时）失败的请求
        this.timer.scheduleAtFixedRate(new TimerTask() {
            @Override
            public void run() {
                try {
                    RequestFutureTable.scanExpiredRequest();
                } catch (Throwable e) {
                    log.error("scan RequestFutureTable exception", e);
                }
            }
        }, 1000 * 3, 1000);
    }

MQClientInstance#registerProducer

public boolean registerProducer(final String group, final DefaultMQProducerImpl producer) {
        if (null == group || null == producer) {
            return false;
        }
 
        MQProducerInner prev = this.producerTable.putIfAbsent(group, producer);
        if (prev != null) {
            log.warn("the producer group[{}] exist already.", group);
            return false;
        }
 
        return true;
    }

同一个JVM同一个生产者组只能有一个生产者实例

疑问：为什么消费者实例start/shutdown方法使用synchronized修饰了，而生产者的start/shutdown方法没有？

目前网上也没搜到相关的问题，去GitHub上提问，目前只得到这两个回答：
1）第一个人是项目的贡献者，说没有明确的相关的文档，自己也倾向于producer的start方法是synchronized的；
2）第二个大概意思是说，大部分情况下生产者一个就足够生产所有topic的消息了，但是需要很多个消费者去消费多个topic的消息。所以对于消费者需要考虑线程安全。

对于2）的回答，我认为也有点牵强。既然加了同步锁，那就证明多线程下有资源争抢。如果是多个消费者，他们是不同的对象（消费者每次都是new出来的），当然也不存在竟态条件。

同样的也注意到DefaultMQPushConsumerImpl的private volatile ServiceState serviceState = ServiceState.CREATE_JUST是加了volatile，而DefaultMQProducerImpl的
private ServiceState serviceState = ServiceState.CREATE_JUST没加，这也是保证多线程环境下的可见性。

综上所述，消费者的多线程调用start方法的场景是什么？多线程环境下有什么问题（自测就算不加synchronized，也会因为对ServiceState的判断而直接抛出异常）？

总结

生产者启动原理：

• 检查生产者配置
• 获取MQClientInstance
• 注册producer
• 同步的方式向所有broker发送心跳
• 启动定时任务处理DefaultMQProducer#request 发送request message（测试发送延时）失败的请求

生产者发送原理

消息发送常用的有三种方式：

• SendResult send(Message msg)，同步发送消息，发送过程完成才返回；
• send(Message msg,SendCallbacksendCallback)，异步发送消息，提供一个回调函数。消息发送出去立马返回，发送过程完成会调用回调函数； public void
• sendOneway(Message msg)，单向发送消息，使用UDP协议，不管发送成功还是失败，即不会等待broker的ack响应。有最大丢失消息的风险；

生产者发送基本流程

源码解析

DefaultMQProducerImpl#sendDefaultImpl()

/**
 * 默认发送消息
 * @param msg
 * @param communicationMode 发送方式，同步、异步、oneway
 * @param sendCallback
 * @param timeout
 * @return
 * @throws MQClientException
 * @throws RemotingException
 * @throws MQBrokerException
 * @throws InterruptedException
 */
private SendResult sendDefaultImpl(
    Message msg,
    final CommunicationMode communicationMode,
    final SendCallback sendCallback,
    final long timeout
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
    //确保生产者运行状态正常
    this.makeSureStateOK();
    //校验消息，包括消息是否为空，topic命名格式，消息体大小是否超过限制（4M）
    Validators.checkMessage(msg, this.defaultMQProducer);
    final long invokeID = random.nextLong();
    long beginTimestampFirst = System.currentTimeMillis();
    long beginTimestampPrev = beginTimestampFirst;
    long endTimestamp = beginTimestampFirst;
    //查找路由信息
    TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
    if (topicPublishInfo != null && topicPublishInfo.ok()) {
        boolean callTimeout = false;
        MessageQueue mq = null;
        Exception exception = null;
        SendResult sendResult = null;
        //重试次数，异步方式发送失败不进行重试
        int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
        int times = 0;
        String[] brokersSent = new String[timesTotal];
        for (; times < timesTotal; times++) {
            String lastBrokerName = null == mq ? null : mq.getBrokerName();
            //选择一个消息队列
            MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
            if (mqSelected != null) {
                mq = mqSelected;
                brokersSent[times] = mq.getBrokerName();
                try {
                    beginTimestampPrev = System.currentTimeMillis();
                    if (times > 0) {
                        //Reset topic with namespace during resend.
                        msg.setTopic(this.defaultMQProducer.withNamespace(msg.getTopic()));
                    }
                    long costTime = beginTimestampPrev - beginTimestampFirst;
                    if (timeout < costTime) {
                        callTimeout = true;
                        break;
                    }
 
                    //真正发送消息
                    sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
                    endTimestamp = System.currentTimeMillis();
                    //更新故障延时机制数据
                    this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
                    switch (communicationMode) {
                        case ASYNC:
                            return null;
                        case ONEWAY:
                            return null;
                        case SYNC:
                            if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
                                if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
                                    continue;
                                }
                            }
 
                            return sendResult;
                        default:
                            break;
                    }
                } catch (RemotingException e) {
                    endTimestamp = System.currentTimeMillis();
                    this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
                    log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
                    log.warn(msg.toString());
                    exception = e;
                    continue;
                } catch (MQClientException e) {
                    endTimestamp = System.currentTimeMillis();
                    this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
                    log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
                    log.warn(msg.toString());
                    exception = e;
                    continue;
                } catch (MQBrokerException e) {
                    endTimestamp = System.currentTimeMillis();
                    this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
                    log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
                    log.warn(msg.toString());
                    exception = e;
                    switch (e.getResponseCode()) {
                        case ResponseCode.TOPIC_NOT_EXIST:
                        case ResponseCode.SERVICE_NOT_AVAILABLE:
                        case ResponseCode.SYSTEM_ERROR:
                        case ResponseCode.NO_PERMISSION:
                        case ResponseCode.NO_BUYER_ID:
                        case ResponseCode.NOT_IN_CURRENT_UNIT:
                            continue;
                        default:
                            if (sendResult != null) {
                                return sendResult;
                            }
 
                            throw e;
                    }
                } catch (InterruptedException e) {
                    endTimestamp = System.currentTimeMillis();
                    this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
                    log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
                    log.warn(msg.toString());
 
                    log.warn("sendKernelImpl exception", e);
                    log.warn(msg.toString());
                    throw e;
                }
            } else {
                break;
            }
        }
 
        if (sendResult != null) {
            return sendResult;
        }
 
        String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
            times,
            System.currentTimeMillis() - beginTimestampFirst,
            msg.getTopic(),
            Arrays.toString(brokersSent));
 
        info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);
 
        MQClientException mqClientException = new MQClientException(info, exception);
        if (callTimeout) {
            throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
        }
 
        if (exception instanceof MQBrokerException) {
            mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
        } else if (exception instanceof RemotingConnectException) {
            mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
        } else if (exception instanceof RemotingTimeoutException) {
            mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
        } else if (exception instanceof MQClientException) {
            mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
        }
 
        throw mqClientException;
    }
 
    //校验namesrv列表是否为空
    validateNameServerSetting();
 
    throw new MQClientException("No route info of this topic: " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
        null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
}

1、消息验证

1. 验证消息对象是否为空；
2. 验证消息body是否为空；
3. 验证topic命名是否规范；
4. 验证消息body长度是否超过4M的限制；

2、查找topic路由信息

/**
 * 查找路由信息
 * @param topic
 * @return
 */
private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
    //先找缓存
    TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
    if (null == topicPublishInfo || !topicPublishInfo.ok()) {
        this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
        //从namesrv获取
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
        topicPublishInfo = this.topicPublishInfoTable.get(topic);
    }
 
    if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
        return topicPublishInfo;
    } else {
        //namesrv也没有路由信息，用默认topic：TBW102路由构造topic信息
        this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
        topicPublishInfo = this.topicPublishInfoTable.get(topic);
        return topicPublishInfo;
    }
}

3、选择一个消息队列

上一步查找到的路由数据结构如下：

TopicPublishInfo：
    //该消息是否是顺序消息
    private boolean orderTopic = false;
    //消息队列列表
    private List<MessageQueue> messageQueueList = new ArrayList<MessageQueue>();
    //每选择一次消息队列，该值会+1，用于选择消息队列
    private volatile ThreadLocalIndex sendWhichQueue = new ThreadLocalIndex();
    private TopicRouteData topicRouteData;
TopicRouteData：
    //topic队列元数据
    private List<QueueData> queueDatas;
    //topic分布的broker元数据
    private List<BrokerData> brokerDatas;

举例说明，如果topicA在broker-a，broker-b上分别创建了4个对列，那么messageQueueList形如：

[
    {"brokerName":"broker-a","queueId":0},
    {"brokerName":"broker-a","queueId":1},
    {"brokerName":"broker-a","queueId":2},
    {"brokerName":"broker-a","queueId":3},
    {"brokerName":"broker-b","queueId":0},
    {"brokerName":"broker-b","queueId":1},
    {"brokerName":"broker-b","queueId":2},
    {"brokerName":"broker-b","queueId":3}
]

如果没开启故障延时机制，则根据messageQueueList轮询
如果开启了故障延时机制（sendLatencyFaultEnable = true，默认false），则会根据broker是否可用将不可用的broker从messageQueueList剔除一定时间。
这个机制的目的是因为：
1）broker不可用以后，上一次选择的queueId是0，发送失败则下一次选择到queueId为1时还是会发送失败，而namesrv检测到broker不可用会延时10s（定时任务检测频率）。
2）namesrv检测到broker下线不会主动告诉生产者，生产者需要通过定时任务（30s）从namesrv拉取topic路由信息时才会感知到。
这种机制保证broker不可用以后，第一次发送消息失败，就将其排除在消息队列的选择范围中。

4、消息发送

在MQClientInstance里面构造netty请求，将生产者组、topic名称、消息发送时间、消息体、RequestCode（请求类型）等信息发送到broker。