RocketMQ之NameServer详解

---

前言

RocketMQ是阿里巴巴开源的一个顶级项目，高性能加上几乎能做到零丢失率让它在越来越多的企业项目中运用，这篇文章不会讲RocketMQ在项目中去如何使用，而是基于源码级别进行剖析讲解。首先来看看RocketMQ的架构图。

RocketMQ分为四个部分：
1、NameServer
NameServer是RocketMQ的路由中心，用于Broker服务的注册和发现，类似zookeeper,但比zookeeper性能更好，实现也更简单，因为NameServer集群中的服务器都是独立的，各实例间相互不进行信息通讯。CAP理论中实现了AP，弱一致性。这让我想到了Eureka，Eureka集群也是实现了AP，但是Eureka server 之间是相互通信的，Eureka client 注册到一个 Eureka server 上就行了，Eureka server 之间会进行同步。而NameServer不用考虑相互间通信，实现起来更简单。
2、Broker
Broker主要负责消息的存储、投递和查询以及服务高可用保证。
3、Producer
消息的生产者。
4、Consumer
消息的消费者。
这篇文章主要讲解NameServer以下内容：
1、NameServer启动过程
2、NameServer路由注册
3、NameServer故障剔除
4、NameServer路由发现

一、NameServer的作用

NameServer是RocketMQ的“大脑”。
路由注册：
所有的Broker实例信息都需要注册到每一个NameServer服务中，就相当于一个Broker需要与NameServer集群中的每一台服务器都要保持连接。
路由发现：
Client(Producer和Consumer)只需要随机选择NameServer集群中一台服务器保持连接，通过这一个NameServer实例获取到Broker实例信息再与Broker进行连接。

二、NameServer启动过程

NameServer启动过程主要包括：
1、填充NameServerConfig、NettyServerConfig属性值。
2、创建NameServerController并初始化。

1.启动类NamesrvStartup

启动方法：

public static NamesrvController main0(String[] args) {
    try {
    	// 创建NamesrvController实例，是NameServer的核心控制器。
        NamesrvController controller = createNamesrvController(args);
        // 启动
        start(controller);
        return controller;
    } catch (Throwable e) {
        e.printStackTrace();
        System.exit(-1);
    }

    return null;
}

createNamesrvController方法主要是创建NamesrvController对象，并配置NameServerConfig、NettyServerConfig属性。

public static NamesrvController createNamesrvController(String[] args) throws IOException, JoranException {
        // 填充NameServerConfig、NettyServerConfig的值
        final NamesrvConfig namesrvConfig = new NamesrvConfig();
        final NettyServerConfig nettyServerConfig = new NettyServerConfig();
        nettyServerConfig.setListenPort(9876);
        // -c configFile 通过-c命令指定配置文件的路径
        // 使用"--属性名 属性值"，例如 --listenPort 9876
        if (commandLine.hasOption('c')) {
            String file = commandLine.getOptionValue('c');
            if (file != null) {
                InputStream in = new BufferedInputStream(new FileInputStream(file));
                properties = new Properties();
                properties.load(in);
                MixAll.properties2Object(properties, namesrvConfig);
                MixAll.properties2Object(properties, nettyServerConfig);
                namesrvConfig.setConfigStorePath(file);
                in.close();
            }
        }
        // -p 打印配置参数
        if (commandLine.hasOption('p')) {
            InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
            MixAll.printObjectProperties(console, namesrvConfig);
            MixAll.printObjectProperties(console, nettyServerConfig);
            System.exit(0);
        }

        MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);
        LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
        JoranConfigurator configurator = new JoranConfigurator();
        configurator.setContext(lc);
        lc.reset();
        configurator.doConfigure(namesrvConfig.getRocketmqHome() + "/conf/logback_namesrv.xml");
        final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);
        // remember all configs to prevent discard
        controller.getConfiguration().registerConfig(properties);
        return controller;
    }

启动

public static NamesrvController start(final NamesrvController controller) throws Exception {
    // 初始化controller
    boolean initResult = controller.initialize();
    //注册JVM钩子函数，优雅的停机方式，在JVM进程关闭之前先将线程池关闭，及时释放资源
    Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, new Callable<Void>() {
        @Override
        public Void call() throws Exception {
            controller.shutdown();
            return null;
        }
    }));
    // 启动核心控制器，以便监听Broker消息生产者的网络请求
    controller.start();
    return controller;
}

controller初始化，主要是加载KV配置创建NettyServer网络处理对象，然后创建两个定时任务：
1、每隔10秒扫描一次有效的Broker，移除无效的Broker。
2、每隔10分钟打印一次KV配置参数

    public boolean initialize() {
        // 加载KV配置
        this.kvConfigManager.load();
        // 创建NettyServer网络处理对象
        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);

        this.remotingExecutor =
            Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));

        this.registerProcessor();
        // 每隔10秒扫描一次有效的Broker，移除无效的Broker
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                NamesrvController.this.routeInfoManager.scanNotActiveBroker();
            }
        }, 5, 10, TimeUnit.SECONDS);
        // 每隔10分钟打印一次KV配置参数
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                NamesrvController.this.kvConfigManager.printAllPeriodically();
            }
        }, 1, 10, TimeUnit.MINUTES);
        return true;
    }

三、NameServer路由注册

RocketMQ的路由注册是通过Broker和NameServer的心跳机制实现的。Broker启动时会向NameServer集群每台机器发送心跳包，后续会每隔30秒发送一次心跳包给NameServer集群。

1、路由元信息

NameServer的路由实现类org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager
中存储了路由元信息，路由注册就是注册这些信息到NameServer中。具体如下：

	/**
     * topic-消息队列映射
     * 一个topic有多个队列，一个Broker默认为每个topic创建4个读队列，4个写队列
     * 根据topic查询出相对应的消息队列
     * topicQueueTable:{
     *     "topic1":[
     *     {
     *         "brokerName":"broker-a",
     *         "readQueueNums":4,
     *         "writeQueueNums":4,
     *         "perm":6,
     *         "topicSynFlag:0
     *     },
     *     {
     *         "brokerName":"broker-b",
     *         "readQueueNums":4,
     *         "writeQueueNums":4,
     *         "perm":6,
     *         "topicSynFlag:0
     *     }
     *     ],
     *     "topic other":[]
     * }
     */
    private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
    /**
     * brokerName-Broker基础信息映射
     * 
     * 根据brokerName可以查询出Broker的其他信息
     * brokerAddrTable:{
     *     "broker-a":{
     *         "cluster":"c1",
     *         "brokerName":"broker-a",
     *         "brokerAddrs":{
     *             0:"192.168.56.1:10000", brokerId=0表示Master
     *             1:"192.168.56.2:10000"  大于0表示Slave
     *         }
     *     },
     *      "broker-a":{
     *         "cluster":"c1",
     *         "brokerName":"broker-b",
     *         "brokerAddrs":{
     *             0:"192.168.56.1:10000",
     *             1:"192.168.56.2:10000"
     *         }
     *     }
     * }
     */
    private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
    /**
     * 集群名称-Broker名称映射，根据集群名称可以查询该集群中所有Broker
     * clusterAddrTable:{
     *     "c1":["broker-a","broker-b"]
     * }
     */
    private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
    /**
     * brokerIP-Broker状态信息映射
     * brokerLiveTable:{
     *     "192.168.56.1:10000":{
     *         "lastUpdateTimestamp":1518270318980,
     *         "dataVersion":versionObj,
     *         "channel":channelObj,
     *         "haServerAddr":""
     *     }
     *      "192.168.56.2:10000":{
     *         "lastUpdateTimestamp":1518270318980,
     *         "dataVersion":versionObj,
     *         "channel":channelObj,
     *         "haServerAddr":""
     *     }
     *      "192.168.56.3:10000":{
     *         "lastUpdateTimestamp":1518270318980,
     *         "dataVersion":versionObj,
     *         "channel":channelObj,
     *         "haServerAddr":""
     *     }
     *      "192.168.56.4:10000":{
     *         "lastUpdateTimestamp":1518270318980,
     *         "dataVersion":versionObj,
     *         "channel":channelObj,
     *         "haServerAddr":""
     *     }
     * }
     */
    private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
    /**
     * brokerIP-FilterServer列表映射
     */
    private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;

2、Broker发送心跳包

核心代码如下：
源码在org.apache.rocketmq.broker.BrokerController#registerBrokerAll

// Broker发送心跳包，默认每隔30秒发送一次，最少10秒，最多60秒
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
        } catch (Throwable e) {
            log.error("registerBrokerAll Exception", e);
        }
    }
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);

组装请求头和请求体，然后循环遍历每一个NameServer进行注册。

brokerAddr：broker 地址。

broker Id：brokerld，0：Master，大于0： Slave 。

brokerName： broker 名称。

clusterName： 集群名称。

haServerAddr：master 地址，初次请求时该值为空， slave 向Nameserver 注册后返回。

requestBody:

• filterServerList 。消息过滤服务器列表。

• topicConfigWrapper。主题配置，topicConfigWrapper 内部封装的是TopicConfigManager中的topicConfigTable，内部存储的是Broker启动时默认的一些Topic，MixAll. SELF_TEST_TOPIC、MixAll.DEFAULT_TOPIC ( AutoCreateTopicEnable=true )., MixAll.BENCHMARK_TOPIC 、MixAll.OFFSET_MOVEDEVENT 、BrokerConfig#brokerClusterName 、BrokerConfig#brokerName 。Broker中Topic 默认存储在${Rocket一Home}/store/config/topic. json 中。
  关于topicConfigTable中的数据可以看看org.apache.rocketmq.broker.topic.TopicConfigManager#TopicConfigManager(org.apache.rocketmq.broker.BrokerController)这个方法代码

	// 组装请求头和请求体
	final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
	requestHeader.setBrokerAddr(brokerAddr);
	requestHeader.setBrokerId(brokerId);
	requestHeader.setBrokerName(brokerName);
	requestHeader.setClusterName(clusterName);
	requestHeader.setHaServerAddr(haServerAddr);
	requestHeader.setCompressed(compressed);
	
	RegisterBrokerBody requestBody = new RegisterBrokerBody();
	requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
	requestBody.setFilterServerList(filterServerList);
	final byte[] body = requestBody.encode(compressed);
	
	for (final String namesrvAddr : nameServerAddressList) {
	    brokerOuterExecutor.execute(new Runnable() {
	        @Override
	        public void run() {
	            try {
	                RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
	                if (result != null) {
	                    registerBrokerResultList.add(result);
	                }
	
	                log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
	            } catch (Exception e) {
	                log.warn("registerBroker Exception, {}", namesrvAddr, e);
	            } finally {
	                countDownLatch.countDown();
	            }
	        }
	    });
	}

3、NameServer处理心跳包

NameServer会通过org.apache.rocketmq.namesrv.processor.DefaultRequestProcessor#processRequest这个方法处理网络请求，源码如下：

public RemotingCommand processRequest(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    switch (request.getCode()) {
        case RequestCode.PUT_KV_CONFIG:
            return this.putKVConfig(ctx, request);
        case RequestCode.GET_KV_CONFIG:
            return this.getKVConfig(ctx, request);
        case RequestCode.DELETE_KV_CONFIG:
            return this.deleteKVConfig(ctx, request);
        case RequestCode.QUERY_DATA_VERSION:
            return queryBrokerTopicConfig(ctx, request);
        case RequestCode.REGISTER_BROKER:
            // 注册Broker
            Version brokerVersion = MQVersion.value2Version(request.getVersion());
            if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
                return this.registerBrokerWithFilterServer(ctx, request);
            } else {
                return this.registerBroker(ctx, request);
            }
        case RequestCode.UNREGISTER_BROKER:
            return this.unregisterBroker(ctx, request);
        case RequestCode.GET_ROUTEINFO_BY_TOPIC:
            // 获取主题的路由信息
            return this.getRouteInfoByTopic(ctx, request);
        case RequestCode.GET_BROKER_CLUSTER_INFO:
            return this.getBrokerClusterInfo(ctx, request);
        case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
            return this.wipeWritePermOfBroker(ctx, request);
        case RequestCode.GET_ALL_TOPIC_LIST_FROM_NAMESERVER:
            return getAllTopicListFromNameserver(ctx, request);
        case RequestCode.DELETE_TOPIC_IN_NAMESRV:
            return deleteTopicInNamesrv(ctx, request);
        case RequestCode.GET_KVLIST_BY_NAMESPACE:
            return this.getKVListByNamespace(ctx, request);
        case RequestCode.GET_TOPICS_BY_CLUSTER:
            return this.getTopicsByCluster(ctx, request);
        case RequestCode.GET_SYSTEM_TOPIC_LIST_FROM_NS:
            return this.getSystemTopicListFromNs(ctx, request);
        case RequestCode.GET_UNIT_TOPIC_LIST:
            return this.getUnitTopicList(ctx, request);
        case RequestCode.GET_HAS_UNIT_SUB_TOPIC_LIST:
            return this.getHasUnitSubTopicList(ctx, request);
        case RequestCode.GET_HAS_UNIT_SUB_UNUNIT_TOPIC_LIST:
            return this.getHasUnitSubUnUnitTopicList(ctx, request);
        case RequestCode.UPDATE_NAMESRV_CONFIG:
            return this.updateConfig(ctx, request);
        case RequestCode.GET_NAMESRV_CONFIG:
            return this.getConfig(ctx, request);
        default:
            break;
    }
    return null;
}

之后会调用org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager#registerBroker这个方法，这个方法主要是填充之前介绍的路由元信息：
clusterAddrTable – Broker集群与Broker映射
brokerAddrTable – Broker名称与Broker基础信息映射
topicQueueTable –
brokerLiveTable
代码如下：

public RegisterBrokerResult registerBroker(
						    final String clusterName,
						    final String brokerAddr,
						    final String brokerName,
						    final long brokerId,
						    final String haServerAddr,
						    final TopicConfigSerializeWrapper topicConfigWrapper,
						    final List<String> filterServerList,
						    final Channel channel) {
    RegisterBrokerResult result = new RegisterBrokerResult();
    try {
        try {
        	// 加写锁，防并发修改
            this.lock.writeLock().lockInterruptibly();
            // 填充clusterAddrTable
            Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
            // borker集群不存在，则创建集群
            if (null == brokerNames) {
                brokerNames = new HashSet<String>();
                this.clusterAddrTable.put(clusterName, brokerNames);
            }
            brokerNames.add(brokerName);

            boolean registerFirst = false;
            // 填充brokerAddrTable
            BrokerData brokerData = this.brokerAddrTable.get(brokerName);
            if (null == brokerData) {
                registerFirst = true;
                brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
                this.brokerAddrTable.put(brokerName, brokerData);
            }
            Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
            //Switch slave to master: first remove <1, IP:PORT> in namesrv, then add <0, IP:PORT>
            //The same IP:PORT must only have one record in brokerAddrTable
            Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
            while (it.hasNext()) {
                Entry<Long, String> item = it.next();
                if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
                    it.remove();
                }
            }

            String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
            registerFirst = registerFirst || (null == oldAddr);

            if (null != topicConfigWrapper
                && MixAll.MASTER_ID == brokerId) {// 如果是master,即brokerId=0
                if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                    || registerFirst) {
                    // 维护topicQueueTable
                    ConcurrentMap<String, TopicConfig> tcTable =
                        topicConfigWrapper.getTopicConfigTable();
                    if (tcTable != null) {
                        for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
                            this.createAndUpdateQueueData(brokerName, entry.getValue());
                        }
                    }
                }
            }
            // 填充brokerLiveTable
            BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                new BrokerLiveInfo(
                    System.currentTimeMillis(), // 放的是当前时间
                    topicConfigWrapper.getDataVersion(),
                    channel,
                    haServerAddr));
            if (null == prevBrokerLiveInfo) {
                log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
            }

            if (filterServerList != null) {
                if (filterServerList.isEmpty()) {
                    this.filterServerTable.remove(brokerAddr);
                } else {
                    this.filterServerTable.put(brokerAddr, filterServerList);
                }
            }

            if (MixAll.MASTER_ID != brokerId) {
                String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
                if (masterAddr != null) {
                    BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
                    if (brokerLiveInfo != null) {
                        result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
                        result.setMasterAddr(masterAddr);
                    }
                }
            }
        } finally {
            this.lock.writeLock().unlock();
        }
    } catch (Exception e) {
        log.error("registerBroker Exception", e);
    }

    return result;
}

四、NameServer路由删除

我们在介绍NameServer启动的时候有个定时任务扫描存活Broker。NameServer会每隔10秒扫描一次BrokerLiveInfo，如果lastUpdateTimestamp在当前时间120秒之前，即lastUpdateTimestamp已经有120秒没有更新了，那么就会移除该broker。
我们来跟踪一下代码：

public void scanNotActiveBroker() {
	// 拿到每一个BrokerLiveInfo遍历
    Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<String, BrokerLiveInfo> next = it.next();
        long last = next.getValue().getLastUpdateTimestamp();
        // last与当前时间相差120s,则直接把它remove掉
        if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
            RemotingUtil.closeChannel(next.getValue().getChannel());
            it.remove();
            log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
            // 移除brokerLiveTable、filterServerTable、brokerAddrTable、clusterAddrTable、topicQueueTable相关信息
            this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
        }
    }
}

五、NameServer路由发现

路由发现是非实时的，Broker推送路由信息给NameServer，NameServer并不会立马将路由信息推送给客户端，而是由客户端自己定时去拉取。

public RemotingCommand getRouteInfoByTopic(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    final GetRouteInfoRequestHeader requestHeader =
        (GetRouteInfoRequestHeader) request.decodeCommandCustomHeader(GetRouteInfoRequestHeader.class);
    // 根据topic查找路由信息
    TopicRouteData topicRouteData = this.namesrvController.getRouteInfoManager().pickupTopicRouteData(requestHeader.getTopic());

    if (topicRouteData != null) {
        // 如果是顺序消息，则从KVConfig中获取关于顺序消息相关的配置填充路由信息
        if (this.namesrvController.getNamesrvConfig().isOrderMessageEnable()) {
            String orderTopicConf =
                this.namesrvController.getKvConfigManager().getKVConfig(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG,
                    requestHeader.getTopic());
            topicRouteData.setOrderTopicConf(orderTopicConf);
        }

        byte[] content = topicRouteData.encode();
        response.setBody(content);
        response.setCode(ResponseCode.SUCCESS);
        response.setRemark(null);
        return response;
    }

    response.setCode(ResponseCode.TOPIC_NOT_EXIST);
    response.setRemark("No topic route info in name server for the topic: " + requestHeader.getTopic()
        + FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
    return response;
}

---

总结

路由注册：Broker每隔30秒给NameServer发送一次心跳包告诉NameServer自己还活着，然后NameServer会更新路由元信息，特别记录心跳时间BrokerLiveInfo.lastUpdateTimestamp。
路由删除：NameServer会每隔10秒扫描一次BrokerLiveInfo.lastUpdateTimestamp，如果该时间差当前时间120秒，NamerServer就会把该Broker的相关信息删除。
路由发现：客户端自己定时拉取topic的相关信息。

那么问题来了：NameServer剔除无效的Broker至少需要延迟120s，那么在这期间Producer获取到的Borker包含该无效的Broker,发送消息给该Broker就会失败，那么RocketMQ是如何处理的呢？下一篇给出答案。

以上内容基本上来自丁威老师的《RocketMQ技术内幕》一书。