RocketMQ源码学习五:Producer启动

作用

RocketMQ发送者Producer的作用是将消息发送到RocketMQ消息队列中。Producer负责将消息封装成消息对象,并将其发送到指定的主题(Topic)中。发送者可以根据需要选择同步发送或异步发送消息,并可以设置消息的延迟等特性。通过使用Producer,应用程序可以将消息发送到RocketMQ,以供其他消费者进行消费和处理。

源码

首先我们平常怎么使用RocketMQ发送消息的,下边时官网中的普通发送的示例代码。

public class SyncProducer {
  public static void main(String[] args) throws Exception {
    // 初始化一个producer并设置Producer group name
    DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name"); //(1)
    // 设置NameServer地址
    producer.setNamesrvAddr("localhost:9876");  //(2)
    // 启动producer
    producer.start();
      Message msg = new Message("TopicTest" /* Topic */,
        "TagA" /* Tag */,`在这里插入代码片`
        ("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body*/
      // 利用producer进行发送,并同步等待发送结果
      SendResult sendResult = producer.send(msg);   //(4)
      System.out.printf("%s%n", sendResult);
    }
    // 一旦producer不再使用,关闭producer
    producer.shutdown();
  }
}

可以看到启动producer的代码producer.start(),今天就学习producer的启动逻辑。入口时DefaultMQProducer.start(),最主要的逻辑时this.defaultMQProducerImpl.start()这段。我们继续向下边看

    public void start(final boolean startFactory) throws MQClientException {
        switch (this.serviceState) {
            case CREATE_JUST:
                this.serviceState = ServiceState.START_FAILED;
                //校验producerGroup
                this.checkConfig();

                if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
                    this.defaultMQProducer.changeInstanceNameToPID();
                }
                //创建个MQClientInstance
                this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook);

                //把当前的生产者实例注册到producerTable 生产者注册表中
                boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
                if (!registerOK) {
                    this.serviceState = ServiceState.CREATE_JUST;
                    throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
                        + "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
                        null);
                }

                //因为刚启动,所以tpic信息为空
                this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());

                //启动生产者相关信息
                if (startFactory) {
                    mQClientFactory.start();
                }

                log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
                    this.defaultMQProducer.isSendMessageWithVIPChannel());
                this.serviceState = ServiceState.RUNNING;
                break;
            case RUNNING:
            case START_FAILED:
            case SHUTDOWN_ALREADY:
                throw new MQClientException("The producer service state not OK, maybe started once, "
                    + this.serviceState
                    + FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
                    null);
            default:
                break;
        }

        //发送心跳信息到所有broker
        this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();

        //删除超时请求,当请求超过设定时间还没有执行,则为超时请求,删除
        this.timer.scheduleAtFixedRate(new TimerTask() {
            @Override
            public void run() {
                try {
                    RequestFutureTable.scanExpiredRequest();
                } catch (Throwable e) {
                    log.error("scan RequestFutureTable exception", e);
                }
            }
        }, 1000 * 3, 1000);
    }

这段逻辑注释中已解释清除,我们还是看主要逻辑:mQClientFactory.start()

    public void start() throws MQClientException {

        synchronized (this) {
            switch (this.serviceState) {
                case CREATE_JUST:
                    //先把服务器状态置为START_FAILED,防止重复启动
                    this.serviceState = ServiceState.START_FAILED;
                    // If not specified,looking address from name server
                    if (null == this.clientConfig.getNamesrvAddr()) {
                        //拉取nameServerAddr地址
                        this.mQClientAPIImpl.fetchNameServerAddr();
                    }
                    // Start request-response channel
                    this.mQClientAPIImpl.start();
                    // Start various schedule tasks
                    //各种定时任务,主要有:发送心跳信息,拉取nameServer地址信息,拉取topic信息
                    this.startScheduledTask();
                    // Start pull service 启动拉消息线程
                    this.pullMessageService.start();
                    // Start rebalance service,启动负载均衡线程,consumer服务数量发生改变,需要再平衡topic消费
                    this.rebalanceService.start();
                    // Start push service  再次调用生产者启动类,确保启动成功
                    this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
                    log.info("the client factory [{}] start OK", this.clientId);
                    //服务状态设置为运行中
                    this.serviceState = ServiceState.RUNNING;
                    break;
                case START_FAILED:
                    throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
                default:
                    break;
            }
        }
    }

这里的逻辑也不算多,主要时启动各种线程和各种定时任务,我们挑两个逻辑仔细查看。一个是:拉取nameServer地址的;另一个是:启动各种定时任务的。这两个逻辑主要是讲topic路由信息的,后期消息发送需要这方面的知识。

nameServerAddr地址的拉取

nameSrvAddr拉取后是有一个和本地nameSrvAddr比较的,如果不同会更新本地nameSrvAddr。

    public final String fetchNSAddr(boolean verbose, long timeoutMills) {
        String url = this.wsAddr;
        try {
            if (!UtilAll.isBlank(this.unitName)) {
                url = url + "-" + this.unitName + "?nofix=1";
            }
            //使用http请求获取nameSrvAddr地址
            HttpTinyClient.HttpResult result = HttpTinyClient.httpGet(url, null, null, "UTF-8", timeoutMills);
            if (200 == result.code) {
                String responseStr = result.content;
                if (responseStr != null) {
                    return clearNewLine(responseStr);
                } else {
                    log.error("fetch nameserver address is null");
                }
            } else {
                log.error("fetch nameserver address failed. statusCode=" + result.code);
            }
        } catch (IOException e) {
            if (verbose) {
                log.error("fetch name server address exception", e);
            }
        }

        if (verbose) {
            String errorMsg =
                "connect to " + url + " failed, maybe the domain name " + MixAll.getWSAddr() + " not bind in /etc/hosts";
            errorMsg += FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL);

            log.warn(errorMsg);
        }
        return null;
    }

上边这段就是拉取获取nameSrvAddr地址的逻辑,可以看到底层他是通过http方式获取nameSrvAddr的。那就有个疑问,这个url是从哪里来的呢?
看这段代码String url = this.wsAddr;继续向下走,可以看到这个wsAddr是对象TopAddressing的属性。找到有参构造代码的地方,可以看到在MQClientAPIImpl对象初始化的时候有对TopAddressing对象初始化。

    public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
        final ClientRemotingProcessor clientRemotingProcessor,
        RPCHook rpcHook, final ClientConfig clientConfig) {
        this.clientConfig = clientConfig;
        topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());

查看getWSAddr()方法

    public static String getWSAddr() {
        String wsDomainName = System.getProperty("rocketmq.namesrv.domain", DEFAULT_NAMESRV_ADDR_LOOKUP);
        String wsDomainSubgroup = System.getProperty("rocketmq.namesrv.domain.subgroup", "nsaddr");
        String wsAddr = "http://" + wsDomainName + ":8080/rocketmq/" + wsDomainSubgroup;
        if (wsDomainName.indexOf(":") > 0) {
            wsAddr = "http://" + wsDomainName + "/rocketmq/" + wsDomainSubgroup;
        }
        return wsAddr;
    }

可以看到这个nameSrvAdd地址是拿的系统变量。这个调用

  1. 到系统变量中拿到nameServer服务地址
  2. 根据地址通过http请求获取nameSrvAddr地址信息
  3. 和本地nameSrvAddr地址信息比较,不同则更新本地nameSrvAddr地址信息
定时任务

producer启动中的定时任务主要是这几个,拉取nameServer地址,拉取最新的topic路由信息,
发送心跳和更新filterServer,清除下线的broker,发送本地缓存的消费进度到broker,调整DefaultMQPushConsumer核心线程数。
其中获取nameServer地址的逻辑上边已经讲过了,逻辑是一样的。下面我们主要讲两个逻辑,一个是拉取topic路由消息,另一个是发送心跳逻辑

topic路由信息的更新
        //延迟10ms,每30秒获取最新的topic信息,新老不同,更新本地的topic信息
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                try {
                    MQClientInstance.this.updateTopicRouteInfoFromNameServer();
                } catch (Exception e) {
                    log.error("ScheduledTask updateTopicRouteInfoFromNameServer exception", e);
                }
            }
        }, 10, this.clientConfig.getPollNameServerInterval(), TimeUnit.MILLISECONDS);

延迟10ms,每30秒获取最新的topic信息,新老不同,更新本地的topic信息。进到代码里

    //获取最新的topic路由信息,并
    public void updateTopicRouteInfoFromNameServer() {
        Set<String> topicList = new HashSet<String>();

        // Consumer  遍历consumerTable中订阅关系的topic信息
        {
            Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
            while (it.hasNext()) {
                Entry<String, MQConsumerInner> entry = it.next();
                MQConsumerInner impl = entry.getValue();
                if (impl != null) {
                    Set<SubscriptionData> subList = impl.subscriptions();
                    if (subList != null) {
                        for (SubscriptionData subData : subList) {
                            topicList.add(subData.getTopic());
                        }
                    }
                }
            }
        }

        // Producer 遍历producerTable中已存在的topic信息
        {
            Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
            while (it.hasNext()) {
                Entry<String, MQProducerInner> entry = it.next();
                MQProducerInner impl = entry.getValue();
                if (impl != null) {
                    Set<String> lst = impl.getPublishTopicList();
                    topicList.addAll(lst);
                }
            }
        }

        //拿到本地所有topic信息去获取最新的topic信息
        for (String topic : topicList) {
            this.updateTopicRouteInfoFromNameServer(topic);
        }
    }

可以看到会遍历出本地所有的topic信息,其中包括consumerTable中订阅关系中的topic信息和producerTable中已存在的topic信息。拿到这些topic后会遍历topic获取最新的topic信息
接下里看this.updateTopicRouteInfoFromNameServer(topic)方法

    public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
        DefaultMQProducer defaultMQProducer) {
        try {
            //加锁
            if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
                try {
                    TopicRouteData topicRouteData;
                    if (isDefault && defaultMQProducer != null) {
                        topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
                            1000 * 3);
                        if (topicRouteData != null) {
                            for (QueueData data : topicRouteData.getQueueDatas()) {
                                //设置topic队列数,可以看到读写队列设置数时一样的
                                int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
                                data.setReadQueueNums(queueNums);
                                data.setWriteQueueNums(queueNums);
                            }
                        }
                    } else {
                        //netty通信,查询topic路由信息
                        topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
                    }
                    if (topicRouteData != null) {
                        TopicRouteData old = this.topicRouteTable.get(topic);
                        boolean changed = topicRouteDataIsChange(old, topicRouteData);
                        if (!changed) {
                            changed = this.isNeedUpdateTopicRouteInfo(topic);
                        } else {
                            log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
                        }

                        //最新topic信息和本地信息不同,消息改变,更新本地信息
                        if (changed) {
                            TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();

                            for (BrokerData bd : topicRouteData.getBrokerDatas()) {
                                this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
                            }

                            // Update Pub info
                            {
                                TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
                                publishInfo.setHaveTopicRouterInfo(true);
                                Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
                                while (it.hasNext()) {
                                    Entry<String, MQProducerInner> entry = it.next();
                                    MQProducerInner impl = entry.getValue();
                                    if (impl != null) {
                                        impl.updateTopicPublishInfo(topic, publishInfo);
                                    }
                                }
                            }

                            // Update sub info
                            {
                                Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
                                Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
                                while (it.hasNext()) {
                                    Entry<String, MQConsumerInner> entry = it.next();
                                    MQConsumerInner impl = entry.getValue();
                                    if (impl != null) {
                                        impl.updateTopicSubscribeInfo(topic, subscribeInfo);
                                    }
                                }
                            }
                            log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
                            this.topicRouteTable.put(topic, cloneTopicRouteData);
                            return true;

上边截取的是主要的逻辑,其中核心逻辑是通过netty网络去nameServer去拉取topic信息。这其中的逻辑有这么一段

                            for (QueueData data : topicRouteData.getQueueDatas()) {
                                //设置topic队列数,可以看到读写队列设置数时一样的
                                int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
                                data.setReadQueueNums(queueNums);
                                data.setWriteQueueNums(queueNums);
                            }

这段是设置topic队列数,可以看到读写队列设置数是一样的。总的流程是这样的

  1. 遍历出本地所有的topic信息,其中包括consumerTable中订阅关系中的topic信息和producerTable中已存在的topic信息
  2. 遍历本地topic信息,根据topic信息去nameServer拉取最新的topic信息
  3. 比较本地topic信息和最新的topic信息,不同的话更新本地topic信息为最新topic信息
发送心跳信息和清除下线的broker信息
        //延迟1s,每30秒获取发送心跳信息和更新filterServer
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                try {
                    //清除下线的broker
                    MQClientInstance.this.cleanOfflineBroker();
                    //向所有broker发送心跳信息
                    MQClientInstance.this.sendHeartbeatToAllBrokerWithLock();
                } catch (Exception e) {
                    log.error("ScheduledTask sendHeartbeatToAllBroker exception", e);
                }
            }
        }, 1000, this.clientConfig.getHeartbeatBrokerInterval(), TimeUnit.MILLISECONDS);

这个定时任务不止是向所有broker发送心跳信息,还包括清除下线的broker和更新filterServer
我们首先看清除下线的broker,代码比较简单就不贴上来了,主要是做了一下逻辑

  1. 到注册表brokerAddrTable中拿到所有的broker地址信息
  2. 遍历注册表中的broker地址
  3. 最新的topicRouteTable注册表中的broker信息是否包含老的broker地址信息
  4. 如果不包含,删除brokerAddrTable中老的broker信息

发送心跳信息到所有broker

            Iterator<Entry<String, HashMap<Long, String>>> it = this.brokerAddrTable.entrySet().iterator();
            while (it.hasNext()) {
                Entry<String, HashMap<Long, String>> entry = it.next();
                String brokerName = entry.getKey();
                HashMap<Long, String> oneTable = entry.getValue();
                if (oneTable != null) {
                    for (Map.Entry<Long, String> entry1 : oneTable.entrySet()) {
                        Long id = entry1.getKey();
                        String addr = entry1.getValue();
                        if (addr != null) {
                            if (consumerEmpty) {
                                //只向master发送心跳
                                if (id != MixAll.MASTER_ID)
                                    continue;
                            }

                            try {
                                int version = this.mQClientAPIImpl.sendHearbeat(addr, heartbeatData, 3000);
                                if (!this.brokerVersionTable.containsKey(brokerName)) {
                                    this.brokerVersionTable.put(brokerName, new HashMap<String, Integer>(4));
                                }
                                this.brokerVersionTable.get(brokerName).put(addr, version);
                                if (times % 20 == 0) {
                                    log.info("send heart beat to broker[{} {} {}] success", brokerName, id, addr);
                                    log.info(heartbeatData.toString());
                                }
                            } catch (Exception e) {

上边是发送心跳的核心逻辑,发送心跳只向master发送心跳,

    public int sendHearbeat(
        final String addr,
        final HeartbeatData heartbeatData,
        final long timeoutMillis
    ) throws RemotingException, MQBrokerException, InterruptedException {
        RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.HEART_BEAT, null);
        request.setLanguage(clientConfig.getLanguage());
        request.setBody(heartbeatData.encode());
        //netty通信,向所有的broker的master发送心跳信息
        RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
        assert response != null;
        switch (response.getCode()) {
            case ResponseCode.SUCCESS: {
                return response.getVersion();
            }
            default:
                break;
        }

        throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
    }

可以看到是使用netty发送心跳信息到broker服务的。同时可以看到返回了Version版本信息,并放入到brokerVersionTable此注册表中。

总结
  1. producer的启动首先也是初始化加载一些信息,比如加载producerGroup,创建MQClientInstance,初始化topicPublishInfoTable注册表
  2. 启动的时候第二部就是拉取nameServer地址,因为后续的topic信息也是需要到nameServer上去拉取的
  3. 启动各种定时任务,包括拉取nameServer地址,topic路由信息的拉取,发送心跳信息等
  4. 同事也会启动PullMessageService线程,这里主要是拉取消息的逻辑,后期会相信学习
  5. 这里边也有一些细节点,比如:topic路由信息里读写队列数量是一样的,只给master发送心跳,各种定时任务的时间设置等,这些对我们平常排查问题还是有很大帮助的

上一篇:RocketMQ源码学习四:Broker启动
下一篇:RocketMQ源码学习六:Producer端消息发送

  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值