RocketMQ源码分析之NameServer

最新推荐文章于 2022-09-20 21:56:50 发布

不爱学习的小妞

最新推荐文章于 2022-09-20 21:56:50 发布

阅读量235

点赞数

分类专栏： rocketmq学习笔记

本文链接：https://blog.csdn.net/qq_25145759/article/details/115553928

版权

rocketmq学习笔记专栏收录该内容

25 篇文章 12 订阅

订阅专栏

前言

在RocketMQ的架构中NameServer起到路由注册中心的作用，它一方面会接收Broker集群的注册信息并将其缓存下来，然后通过心跳监测机制来判断Broker是否存活；一方面Broker在向其发送注册的同时还会将Broker端的Topic及其配置信息发送给NameServer，NameServer也会缓存Topic相关信息。本节将通过NameServer的启动流程、线程通信模型、Broker向NameServer注册路由、NameServer端路由信息都包含什么信息以及什么情况下会触发路由信息被删除来分析NameServer。

一、NameServer启动流程

正如BrokerController在Broker端的作用一样，在NameServer端NamesrvController负责管理各组件的生命周期：初始化、启动和停止。NameServer启动的入口是NamesrvStartup的main方法，主要包含以下三部分：
1.创建NamesrvController

public static NamesrvController createNamesrvController(String[] args) throws IOException, JoranException {
        System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
        //PackageConflictDetect.detectFastjson();

        Options options = ServerUtil.buildCommandlineOptions(new Options());
        commandLine = ServerUtil.parseCmdLine("mqnamesrv", args, buildCommandlineOptions(options), new PosixParser());
        if (null == commandLine) {
            System.exit(-1);
            return null;
        }

        final NamesrvConfig namesrvConfig = new NamesrvConfig();
        final NettyServerConfig nettyServerConfig = new NettyServerConfig();
        nettyServerConfig.setListenPort(9876);
        if (commandLine.hasOption('c')) {
            String file = commandLine.getOptionValue('c');
            if (file != null) {
                InputStream in = new BufferedInputStream(new FileInputStream(file));
                properties = new Properties();
                properties.load(in);
                MixAll.properties2Object(properties, namesrvConfig);
                MixAll.properties2Object(properties, nettyServerConfig);

                namesrvConfig.setConfigStorePath(file);

                System.out.printf("load config properties file OK, %s%n", file);
                in.close();
            }
        }

        if (commandLine.hasOption('p')) {
            InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
            MixAll.printObjectProperties(console, namesrvConfig);
            MixAll.printObjectProperties(console, nettyServerConfig);
            System.exit(0);
        }

        MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);

        if (null == namesrvConfig.getRocketmqHome()) {
            System.out.printf("Please set the %s variable in your environment to match the location of the RocketMQ installation%n", MixAll.ROCKETMQ_HOME_ENV);
            System.exit(-2);
        }

        LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
        JoranConfigurator configurator = new JoranConfigurator();
        configurator.setContext(lc);
        lc.reset();
        configurator.doConfigure(namesrvConfig.getRocketmqHome() + "/conf/logback_namesrv.xml");

        log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);

        MixAll.printObjectProperties(log, namesrvConfig);
        MixAll.printObjectProperties(log, nettyServerConfig);

        final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);

        // remember all configs to prevent discard
        controller.getConfiguration().registerConfig(properties);

        return controller;
    }

创建NamesrvController的流程图如下所示：
在这里插入图片描述
2.初始化NamesrvController

public boolean initialize() {

        this.kvConfigManager.load();

        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);

        this.remotingExecutor =
            Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));

        this.registerProcessor();

        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                NamesrvController.this.routeInfoManager.scanNotActiveBroker();
            }
        }, 5, 10, TimeUnit.SECONDS);

        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

            @Override
            public void run() {
                NamesrvController.this.kvConfigManager.printAllPeriodically();
            }
        }, 1, 10, TimeUnit.MINUTES);

        if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
            // Register a listener to reload SslContext
            try {
                fileWatchService = new FileWatchService(
                    new String[] {
                        TlsSystemConfig.tlsServerCertPath,
                        TlsSystemConfig.tlsServerKeyPath,
                        TlsSystemConfig.tlsServerTrustCertPath
                    },
                    new FileWatchService.Listener() {
                        boolean certChanged, keyChanged = false;
                        @Override
                        public void onChanged(String path) {
                            if (path.equals(TlsSystemConfig.tlsServerTrustCertPath)) {
                                log.info("The trust certificate changed, reload the ssl context");
                                reloadServerSslContext();
                            }
                            if (path.equals(TlsSystemConfig.tlsServerCertPath)) {
                                certChanged = true;
                            }
                            if (path.equals(TlsSystemConfig.tlsServerKeyPath)) {
                                keyChanged = true;
                            }
                            if (certChanged && keyChanged) {
                                log.info("The certificate and private key changed, reload the ssl context");
                                certChanged = keyChanged = false;
                                reloadServerSslContext();
                            }
                        }
                        private void reloadServerSslContext() {
                            ((NettyRemotingServer) remotingServer).loadSslContext();
                        }
                    });
            } catch (Exception e) {
                log.warn("FileWatchService created error, can't load the certificate dynamically");
            }
        }

3.启动NamesrvController

public void start() throws Exception {
        this.remotingServer.start();

        if (this.fileWatchService != null) {
            this.fileWatchService.start();
        }
    }

初始化及启动NamesrvController的流程如下所示：
在这里插入图片描述

二、NameServer通信线程模型

从上面NameServer的启动流程可以看出：
1.NameServer端的通信模型与Broker端是一样的
2.在初始化NamesrvController过程会注册processor，该processor是用来处理收到的请求，从这里可以看出在NameServer端是在remotingExecutor线程池来处理业务
NameServer端的线程模型图如下所示（详细见：RocketMQ源码分析之通信模块）：
在这里插入图片描述

三、Broker向NameServer注册路由

Broker在启动的过程中会主动向NameServer发起注册路由的请求并且还会启动一个定时任务定时向NameServer发送路由信息，这里Broker启动时发送路由注册请求的方法与定时任务中调用的方法是同一个方法。所以这里的入口是registerBrokerAll方法（在Dledger模式中，当选举完成后会在changeToSlave和changeToMaster方法中调用该方法）

if (!messageStoreConfig.isEnableDLegerCommitLog()) {
    startProcessorByHa(messageStoreConfig.getBrokerRole());
    handleSlaveSynchronize(messageStoreConfig.getBrokerRole());
    this.registerBrokerAll(true, false, true);
}

this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        try {
            BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
        } catch (Throwable e) {
            log.error("registerBrokerAll Exception", e);
        }
    }
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);

接着来详细看registerBrokerAll方法。首先会通过buildTopicConfigSerializeWrapper方法将Broker端缓存中的topicConfigTable中topic配置信息以及dataVersion封装成TopicConfigSerializeWrapper对象。然后判断Broker的权限是否是不能写或者不能读，如果是则遍历TopicConfigSerializeWrapper对象的topicConfigTable并修改每条topic配置信息中的权限。最后会先进行一个简单的判断，如果条件成立则会执行doRegisterBrokerAll方法，即向所有的NameServer注册，判断条件是forceRegister || needRegister(…)，如果不是在定时任务中调用则forceRegister的值为true表示在Broker启动过程中第一次必须向NameServer注册路由，后面定时任务中forceRegister的值是broker配置文件中配置项forceRegister的值，needRegister方法的实现是向所有的NameServer发送RequestCode.QUERY_DATA_VERSION请求，如果所有NameServer中的路由信息版本与Broker端数据一致则needRegister返回false，如果所有的NameServer中哪怕有一个节点缓存brokerLiveTable不包含该Broker的信息或者是包含但是版本不一致则needRegister返回true。

public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
        TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();

        if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
            || !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
            ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
            for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
                TopicConfig tmp =
                    new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
                        this.brokerConfig.getBrokerPermission());
                topicConfigTable.put(topicConfig.getTopicName(), tmp);
            }
            topicConfigWrapper.setTopicConfigTable(topicConfigTable);
        }

        if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
            this.getBrokerAddr(),
            this.brokerConfig.getBrokerName(),
            this.brokerConfig.getBrokerId(),
            this.brokerConfig.getRegisterBrokerTimeoutMills())) {
            doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
        }
    }

doRegisterBrokerAll方法主要包含两部分，一个是Broker向所有的NameServer发送RequestCode.REGISTER_BROKER注册请求并等待请求结果，一个是根据该请求结果进行一系列操作。

private void doRegisterBrokerAll(boolean checkOrderConfig, boolean oneway,
        TopicConfigSerializeWrapper topicConfigWrapper) {
        List<RegisterBrokerResult> registerBrokerResultList = this.brokerOuterAPI.registerBrokerAll(
            this.brokerConfig.getBrokerClusterName(),
            this.getBrokerAddr(),
            this.brokerConfig.getBrokerName(),
            this.brokerConfig.getBrokerId(),
            this.getHAServerAddr(),
            topicConfigWrapper,
            this.filterServerManager.buildNewFilterServerList(),
            oneway,
            this.brokerConfig.getRegisterBrokerTimeoutMills(),
            this.brokerConfig.isCompressedRegister());

        if (registerBrokerResultList.size() > 0) {
            RegisterBrokerResult registerBrokerResult = registerBrokerResultList.get(0);
            if (registerBrokerResult != null) {
                if (this.updateMasterHAServerAddrPeriodically && registerBrokerResult.getHaServerAddr() != null) {
                    this.messageStore.updateHaMasterAddress(registerBrokerResult.getHaServerAddr());
                }

                this.slaveSynchronize.setMasterAddr(registerBrokerResult.getMasterAddr());

                if (checkOrderConfig) {
                    this.getTopicConfigManager().updateOrderTopicConfig(registerBrokerResult.getKvTable());
                }
            }
        }
    }

下面详细说下这两点：
1.Broker向所有NameServer发送注册请求及NameServer处理
（1）Broker向所有NameServer发送注册请求

请求是同步调用类型
通过CountDownLatch来控制请求并发，即Broker需要等待该请求的所有结果返回
最终返回的是一个List

public List<RegisterBrokerResult> registerBrokerAll(
        final String clusterName,
        final String brokerAddr,
        final String brokerName,
        final long brokerId,
        final String haServerAddr,
        final TopicConfigSerializeWrapper topicConfigWrapper,
        final List<String> filterServerList,
        final boolean oneway,
        final int timeoutMills,
        final boolean compressed) {

        final List<RegisterBrokerResult> registerBrokerResultList = new CopyOnWriteArrayList<>();
        List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
        if (nameServerAddressList != null && nameServerAddressList.size() > 0) {

            final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
            requestHeader.setBrokerAddr(brokerAddr);
            requestHeader.setBrokerId(brokerId);
            requestHeader.setBrokerName(brokerName);
            requestHeader.setClusterName(clusterName);
            requestHeader.setHaServerAddr(haServerAddr);
            requestHeader.setCompressed(compressed);

            RegisterBrokerBody requestBody = new RegisterBrokerBody();
            requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
            requestBody.setFilterServerList(filterServerList);
            final byte[] body = requestBody.encode(compressed);
            final int bodyCrc32 = UtilAll.crc32(body);
            requestHeader.setBodyCrc32(bodyCrc32);
            final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
            for (final String namesrvAddr : nameServerAddressList) {
                brokerOuterExecutor.execute(new Runnable() {
                    @Override
                    public void run() {
                        try {
                            RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
                            if (result != null) {
                                registerBrokerResultList.add(result);
                            }

                            log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
                        } catch (Exception e) {
                            log.warn("registerBroker Exception, {}", namesrvAddr, e);
                        } finally {
                            countDownLatch.countDown();
                        }
                    }
                });
            }

            try {
                countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
            } catch (InterruptedException e) {
            }
        }

        return registerBrokerResultList;
    }

（2）NameServer处理注册请求

NameServer端由DefaultRequestProcessor来处理注册请求，具体是registerBrokerWithFilterServer方法
在registerBrokerWithFilterServer方法中会调用RouteInfoManager的registerBroker方法来更新NameServer端的路由缓存clusterAddrTable、brokerAddrTable、topicQueueTable、brokerLiveTable和filterServerTable（这里也是下面分析路由信息的入口）
最终该请求返回的结果中包含haServerAddr、masterAddr及 kvConfig

public RemotingCommand registerBrokerWithFilterServer(ChannelHandlerContext ctx, RemotingCommand request)
        throws RemotingCommandException {
        final RemotingCommand response = RemotingCommand.createResponseCommand(RegisterBrokerResponseHeader.class);
        final RegisterBrokerResponseHeader responseHeader = (RegisterBrokerResponseHeader) response.readCustomHeader();
        final RegisterBrokerRequestHeader requestHeader =
            (RegisterBrokerRequestHeader) request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);

        if (!checksum(ctx, request, requestHeader)) {
            response.setCode(ResponseCode.SYSTEM_ERROR);
            response.setRemark("crc32 not match");
            return response;
        }

        RegisterBrokerBody registerBrokerBody = new RegisterBrokerBody();

        if (request.getBody() != null) {
            try {
                registerBrokerBody = RegisterBrokerBody.decode(request.getBody(), requestHeader.isCompressed());
            } catch (Exception e) {
                throw new RemotingCommandException("Failed to decode RegisterBrokerBody", e);
            }
        } else {
            registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setCounter(new AtomicLong(0));
            registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setTimestamp(0);
        }

        RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
            requestHeader.getClusterName(),
            requestHeader.getBrokerAddr(),
            requestHeader.getBrokerName(),
            requestHeader.getBrokerId(),
            requestHeader.getHaServerAddr(),
            registerBrokerBody.getTopicConfigSerializeWrapper(),
            registerBrokerBody.getFilterServerList(),
            ctx.channel());

        responseHeader.setHaServerAddr(result.getHaServerAddr());
        responseHeader.setMasterAddr(result.getMasterAddr());

        byte[] jsonValue = this.namesrvController.getKvConfigManager().getKVListByNamespace(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG);
        response.setBody(jsonValue);

        response.setCode(ResponseCode.SUCCESS);
        response.setRemark(null);
        return response;
    }

2.Broker端处理NameServer端返回的结果
Broker端收到NameServer端返回的注册请求的结果后，如果该List的大小如果大于0则获取结果集中的第一个，首先先判断updateMasterHAServerAddrPeriodically是否为true且结果中的haServerAddr是不为空，如果条件成立则更新DefaultMessageStore的master地址，这里需要注意在Broker初始化的过程中会对updateMasterHAServerAddrPeriodically进行赋值，只针对slave节点设置，默认情况下updateMasterHAServerAddrPeriodically为false，如果当前节点的角色是slave且节点配置文件中haMasterAddr不为空且长度大于等于6则updateMasterHAServerAddrPeriodically为false，否则为true（即salve节点的配置文件中没有设置haMasterAddr）。其次会更新slaveSynchronize的master地址。最后会根据请求结果中的kvConfig来更新broker端的topic配置中的order属性。

四、路由信息详解

分析NameServer端路由信息的入口是RouteInfoManager，这里记录了NameServer端缓存的路由信息，并且还可以看出使用了读写锁来控制对路由信息的读写操作：

public class RouteInfoManager {
    private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
    private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
    private final ReadWriteLock lock = new ReentrantReadWriteLock();
    private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
    private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
    private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
    private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
    private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;

    public RouteInfoManager() {
        this.topicQueueTable = new HashMap<String, List<QueueData>>(1024);
        this.brokerAddrTable = new HashMap<String, BrokerData>(128);
        this.clusterAddrTable = new HashMap<String, Set<String>>(32);
        this.brokerLiveTable = new HashMap<String, BrokerLiveInfo>(256);
        this.filterServerTable = new HashMap<String, List<String>>(256);
    }

1.topicQueueTable
topicQueueTable的数据结构是<topicName, List>，从其结构可以看出里面缓存了topic的队列信息，具体包含topic所在broker的brokerName、读队列数readQueueNums、写队列数writeQueueNums、权限perm及系统标记topicSynFlag

2.brokerAddrTable
brokerAddrTable的数据结构是<brokerName, BrokerData>，从其结构可以看出里面缓存了broker相关的数据，具体内容如下：

集群名称cluster
broker名称brokerName
brokerAddrs，其结构是<brokerId, broker address>，里面记录的是一组broker中所有节点的地址，其中key是brokerId，master的brokerId为0，slave的brokerId大于1

3.clusterAddrTable
clusterAddrTable的数据结构是<cluster, set>，里面记录的是一个集群中所有broker的名称

4.brokerLiveTable
brokerLiveTable的数据结构是<brokerAddr, BrokerLiveInfo>，里面记录的是NameServer端判断的已存活的broker的信息，具体包含：上次更新发送心跳信息的时间lastUpdateTimestamp、数据版本dataVersion、连接channel和ha地址haServerAddr

5.filterServerTable
filterServerTable的数据结构是<brokerAddr, List>，里面记录的是broker上的FilterServer列表

五、删除路由信息

在以下场景下NameServer会删除路由信息：
1.Broker正常关闭的情况下，会向所有的NameServer发送RequestCode.UNREGISTER_BROKER请求通知NameServer自己要下线了，NameServer在收到请求后会删除与此Broker相关的路由信息（这里的入口是Broker端BrokerController的shutdown方法，该方法会调用UNregisterBrokerAll方法）

2.NameServer端在启动的过程会启动一个定时任务：每10秒对brokerLiveTable中的数据进行一次遍历，如果距离更新上次心跳信息的时间大于120秒则会关闭NameServer端与此Broker连接的channel并删除与此Broker相关的路由信息

public void scanNotActiveBroker() {
        Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
        while (it.hasNext()) {
            Entry<String, BrokerLiveInfo> next = it.next();
            long last = next.getValue().getLastUpdateTimestamp();
            if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
                RemotingUtil.closeChannel(next.getValue().getChannel());
                it.remove();
                log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
                this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
            }
        }
    }

不爱学习的小妞

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
RocketMQ源码分析之NameServer

目录前言一、NameServer启动流程二、NameServer通信线程模型三、Broker向NameServer注册路由四、路由信息详解五、删除路由信息前言在RocketMQ的架构中NameServer起到路由注册中心的作用，它一方面会接收Broker集群的注册信息并将其缓存下来，然后通过心跳监测机制来判断Broker是否存活；一方面Broker在向其发送注册的同时还会将Broker端的Topic及其配置信息发送给NameServer，NameServer也会缓存Topic相关信息。本节将通过Nam
复制链接

扫一扫