【rocketmq系列】【三】【NameServer-路由中心】

路由管理，服务注册，服务发现
存储的是什么数据？
避免单点故障，提供高可用？

接口scheduleAtFixedRate原型定义及参数说明

接口scheduleAtFixedRate原型定义及参数说明

 public ScheduledFuture<?> scheduleAtFixedRate(Runnable command,
				long initialDelay,
				long period,
				TimeUnit unit);
command：执行线程
initialDelay：初始化延时
period：两次开始执行最小间隔时间
unit：计时单位

接口scheduleWithFixedDelay原型定义及参数说明

public ScheduledFuture<?> scheduleWithFixedDelay(Runnable command,
				long initialDelay,
				long delay,
				TimeUnit unit);
command：执行线程
initialDelay：初始化延时
period：前一次执行结束到下一次执行开始的间隔时间（间隔执行延迟时间）
unit：计时单位

Broker消息服务器启动时向所有NameServer注册，消息生产者在发送消息之前先从NameServer获取Broker服务器地址列表，然后根据负载均衡算法进行选择一台服务器进行消息发送。

NameServer与每台Broker保持长链接，并间隔30s检测Broker是否存活，如果检测到Broker宕机，就会从路由注册表中移除。但是路由变化不会通知生产者（为了降低NameServer的复杂度），在消息发送端提供容错机制保证消息发送的高可用性。

NameServer本身的高可用通过部署多台服务器来实现，但是彼此之间不通信，NameServer服务器在某一个时刻的数据并不完全相同，这对消息发送没有影响。

路由注册&故障剔除

路由元信息

参数解释

• org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager

// 一个Topic拥有多个消息队列，一个Broker为每一个主题默认创建4个写队列，4个读队列。
// 多个Broker组成一个集群，BrokerName由相同的多台Broker组成Master-Slave结构。
// brokerId为0表示Master，大于0表示Slave。
// BrokerLiveInfo中的lastUpdateTimestamp存储上次收到的Broker心跳包的时间

// Topic消息队列路由信息，消息发送时根据路由表进行负载均衡
private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;

// Broker基础信息，brokerName，集群名称，主备Broker地址
private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;

// Broker集群信息，存储集群中所有的Broker名称
private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;

// Broker状态信息，NameServer每次收到心跳包会更新这个信息
private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;

// Broker上的FilterServer列表，用于类模式消息过滤，
private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;

路由元数据类图

运行时数据结构

路由注册

通过Broker与NameServer的心跳功能实现的。Broker启动时向所有NameServer发送心跳语句，然后每隔30s向集群中所有NameServer发送心跳包，NameServer收到后会更新lastUpdateTimeStamp，然后NameServer每隔10s扫描brokerLiveTable，如果连续120s都没有收到，NameServer将移除该Broker的路由信息同时关闭Socket连接。

Broker启动时会向所有的NameServer发送心跳请求，每隔(10秒到60秒，可通过配置进行配置)秒向集群中所有的NameServer发送心跳包。NameServer收到心跳包时会更新brokerLiveTable表中的BrokerLiveInfo。同时NameServer会每隔10秒扫描一次brokerLiveTable。如果连续120秒都没有收到心跳包，则会移除Broker。

发送心跳包

org.apache.rocketmq.broker.BrokerController#start

this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

     @Override
     public void run() {
         try {
             BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
         } catch (Throwable e) {
             log.error("registerBrokerAll Exception", e);
         }
     }
 }, 1000 * 10
 , Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000))
 , TimeUnit.MILLISECONDS
);

registerBrokerAll方法中会更新Topic消息队列路由信息。后面判断是否需要注册，如果是，则进行注册。
org.apache.rocketmq.broker.BrokerController#registerBrokerAll

处理心跳包

• org.apache.rocketmq.namesrv.processor.DefaultRequestProcessor
  

Version brokerVersion = MQVersion.value2Version(request.getVersion());
if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
	// 新版本都使用这个
    return this.registerBrokerWithFilterServer(ctx, request);
} else {
    return this.registerBroker(ctx, request);
}

核心注册逻辑在org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager#registerBroker实现

1、判断Broker所属集群是否存在，若不存在，则创建。

2、维护BrokerData信息。首先从brokerAddrTable根据BrokerName尝试获取Broker信息，如果不存在，则新建BrokerData并放入到brokerAddrTable，registerFirst设置为true。如果存在，则删除原来的。

3、如果Broker为Master,并且Broker Topic配置信息发生变化或者是初次注册，则需要创建或更新Topic路由数据，填充TopicQueueTable。

4、更新BrokerLiveInfo。

5、注册Broker的过滤器Server地址列表。

public RemotingCommand registerBrokerWithFilterServer(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(RegisterBrokerResponseHeader.class);
    final RegisterBrokerResponseHeader responseHeader = (RegisterBrokerResponseHeader) response.readCustomHeader();
    final RegisterBrokerRequestHeader requestHeader =
        (RegisterBrokerRequestHeader) request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);

    if (!checksum(ctx, request, requestHeader)) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("crc32 not match");
        return response;
    }

    RegisterBrokerBody registerBrokerBody = new RegisterBrokerBody();

    if (request.getBody() != null) {
        try {
            registerBrokerBody = RegisterBrokerBody.decode(request.getBody(), requestHeader.isCompressed());
        } catch (Exception e) {
            throw new RemotingCommandException("Failed to decode RegisterBrokerBody", e);
        }
    } else {
        registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setCounter(new AtomicLong(0));
        registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setTimestamp(0);
    }

    RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
        requestHeader.getClusterName(),
        requestHeader.getBrokerAddr(),
        requestHeader.getBrokerName(),
        requestHeader.getBrokerId(),
        requestHeader.getHaServerAddr(),
        registerBrokerBody.getTopicConfigSerializeWrapper(),
        registerBrokerBody.getFilterServerList(),
        ctx.channel());

    responseHeader.setHaServerAddr(result.getHaServerAddr());
    responseHeader.setMasterAddr(result.getMasterAddr());

    byte[] jsonValue = this.namesrvController.getKvConfigManager().getKVListByNamespace(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG);
    response.setBody(jsonValue);

    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

public RemotingCommand registerBroker(ChannelHandlerContext ctx,
    RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(RegisterBrokerResponseHeader.class);
    final RegisterBrokerResponseHeader responseHeader = (RegisterBrokerResponseHeader) response.readCustomHeader();
    final RegisterBrokerRequestHeader requestHeader =
        (RegisterBrokerRequestHeader) request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);

    if (!checksum(ctx, request, requestHeader)) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("crc32 not match");
        return response;
    }

    TopicConfigSerializeWrapper topicConfigWrapper;
    if (request.getBody() != null) {
        topicConfigWrapper = TopicConfigSerializeWrapper.decode(request.getBody(), TopicConfigSerializeWrapper.class);
    } else {
        topicConfigWrapper = new TopicConfigSerializeWrapper();
        topicConfigWrapper.getDataVersion().setCounter(new AtomicLong(0));
        topicConfigWrapper.getDataVersion().setTimestamp(0);
    }

    RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
        requestHeader.getClusterName(),
        requestHeader.getBrokerAddr(),
        requestHeader.getBrokerName(),
        requestHeader.getBrokerId(),
        requestHeader.getHaServerAddr(),
        topicConfigWrapper,
        null,
        ctx.channel()
    );

    responseHeader.setHaServerAddr(result.getHaServerAddr());
    responseHeader.setMasterAddr(result.getMasterAddr());

    byte[] jsonValue = this.namesrvController.getKvConfigManager().getKVListByNamespace(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG);
    response.setBody(jsonValue);
    response.setCode(ResponseCode.SUCCESS);
    response.setRemark(null);
    return response;
}

public RegisterBrokerResult registerBroker(
    final String clusterName,
    final String brokerAddr,
    final String brokerName,
    final long brokerId,
    final String haServerAddr,
    final TopicConfigSerializeWrapper topicConfigWrapper,
    final List<String> filterServerList,
    final Channel channel) {
    RegisterBrokerResult result = new RegisterBrokerResult();
    try {
        try {
            this.lock.writeLock().lockInterruptibly();

            // Broker集群信息维护，如果不存在就新建
            // 集群名称 映射 broker的名字列表
            Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
            if (null == brokerNames) {
                brokerNames = new HashSet<String>();
                this.clusterAddrTable.put(clusterName, brokerNames);
            }
            brokerNames.add(brokerName);

            boolean registerFirst = false;

            // BrokerData数据维护，如果不存在就新建
            //
            BrokerData brokerData = this.brokerAddrTable.get(brokerName);
            if (null == brokerData) {
                registerFirst = true;
                brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
                this.brokerAddrTable.put(brokerName, brokerData);
            }
            Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
            //Switch slave to master: first remove <1, IP:PORT> in namesrv, then add <0, IP:PORT>
            //The same IP:PORT must only have one record in brokerAddrTable
            Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
            while (it.hasNext()) {
                Entry<Long, String> item = it.next();
                if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
                    it.remove();
                }
            }

            String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
            registerFirst = registerFirst || (null == oldAddr);

            // 如果broker是master
            if (null != topicConfigWrapper
                && MixAll.MASTER_ID == brokerId) {
                // 如果配置信息发生变化 或者 第一次注册，需要创建或更新Topic路由信息
                // 当消息生产者发送主题时，如果该主题未创建，并且BrokerConfig的autoCreateTopicEnable为true时
                // 将返回MixAll.DEFAULT_TOPIC的路由信息
                if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                    || registerFirst) {
                    ConcurrentMap<String, TopicConfig> tcTable =
                        topicConfigWrapper.getTopicConfigTable();
                    if (tcTable != null) {
                        for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
                            this.createAndUpdateQueueData(brokerName, entry.getValue());
                        }
                    }
                }
            }

            BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                new BrokerLiveInfo(
                    System.currentTimeMillis(),
                    topicConfigWrapper.getDataVersion(),
                    channel,
                    haServerAddr));
            if (null == prevBrokerLiveInfo) {
                log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
            }

            if (filterServerList != null) {
                if (filterServerList.isEmpty()) {
                    this.filterServerTable.remove(brokerAddr);
                } else {
                    this.filterServerTable.put(brokerAddr, filterServerList);
                }
            }

            if (MixAll.MASTER_ID != brokerId) {
                String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
                if (masterAddr != null) {
                    BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
                    if (brokerLiveInfo != null) {
                        result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
                        result.setMasterAddr(masterAddr);
                    }
                }
            }
        } finally {
            this.lock.writeLock().unlock();
        }
    } catch (Exception e) {
        log.error("registerBroker Exception", e);
    }

    return result;
}

路由剔除

Rocket MQ有两个触发点来触发删除路由。
1、定时扫描，心跳超过120秒
1、Broker正常下线情况下，会执行移除操作。

org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager#onChannelDestroy

1、申请写锁，根据brokerAddress从brokerLiveTable和filterServerTable移除。

2、维护brokerAddrTable。遍历HashMap_<String/* brokerName /, BrokerData> brokerAddrTable,从HashMap<Long/ brokerId /, String/ broker address */> _brokerAddrs中找到具体的Broker，进行移除。如果移除后在BrokerData中不再包含其他Broker，则在brokerAddrTable中移除该 brokerName对应的条目。

3、根据brokerName,从clusterAddrTable中删除对应的Broker

4、根据brokerName遍历所有的主题队列。

// 定时任务一：每隔10秒扫描一次Broker，删除不活跃的Broker。
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {

    @Override
    public void run() {
        NamesrvController.this.routeInfoManager.scanNotActiveBroker();
    }
}, 5, 10, TimeUnit.SECONDS);


public void scanNotActiveBroker() {
    Iterator<Entry<String, BrokerLiveInfo>> it = this.brokerLiveTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<String, BrokerLiveInfo> next = it.next();
        long last = next.getValue().getLastUpdateTimestamp();
        if ((last + BROKER_CHANNEL_EXPIRED_TIME) < System.currentTimeMillis()) {
            RemotingUtil.closeChannel(next.getValue().getChannel());
            it.remove();
            log.warn("The broker channel expired, {} {}ms", next.getKey(), BROKER_CHANNEL_EXPIRED_TIME);
            this.onChannelDestroy(next.getKey(), next.getValue().getChannel());
        }
    }
}

路由发现