RocketMQ源码 Namesrv-RouteInfoManager路由管理组件

前言

RocketMQ在启动 Namesrv 的过程中会初始化 NamesrvController 核心组件，NamesrvController初始化过程中 又引用了 RouteInfoManager 路由信息管理组件，它负责对管理主题队列和Broker集群路由分布信息。

源码版本：4.9.3

相关数据模型

1个namesrv 对应 N个broker集群；

1个broker集群 对应 N个borker组；

1个broker组 对应 N个broker节点，组成一个HA高可用模式；

1个topic 对应 N个queue队列，单个队列相当于topic的1个数据分片，且队列实际可分布在不同broker组，同一个broker组存储的内存队列数据相同。

核心数据结构

RouteInfoManager 路由信息管理 第一层依赖结构：

public class RouteInfoManager {
    private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
    private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
    private final ReadWriteLock lock = new ReentrantReadWriteLock();
    private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
    private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
    private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
    private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
    private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}

解析：

1. BROKER_CHANNEL_EXPIRED_TIME：Broker长链接空闲过期时间；
2. lock：读写锁，用于读写当前类维护的数据结构时，进行并发控制；
3. topicQueueTable：key-topic，value-队列列表，topic是逻辑概念，内部对应多个queue；
4. brokerAddrTable：key-broker组名，value-HA broker，一个broker组对应了，多个broker节点，组成一个HA高可用架构；
5. clusterAddrTable：key-broker集群名，value-broker组名；一个broker集群对应多个broker组；
6. brokerLiveTable：key-broker节点ip地址，value-broker节点心跳、保活信息；
7. filterServerTable：key-broker节点ip地址，value-filterServer，filterServer是进行消息过滤的高级特性，支持比tag过滤，更为负责的过滤逻辑，可以实现自定义类。在实际消费过程中，broker将自定义类和数据都传给同一机器上部署的 filter Server进程，过滤后，再给consumer消费。

QueueData 队列 数据结构：

public class QueueData implements Comparable<QueueData> {
    private String brokerName;
    private int readQueueNums;
    private int writeQueueNums;
    private int perm;
    private int topicSysFlag;
}

解析：

1. brokerName：队列所在broker组名称；
2. readQueueNums：读队列数量，读、写队列用于支持扩缩容能力；
3. writeQueueNums：写队列数量；
4. perm：权限级别；

BrokerData broker组 数据结构：

public class BrokerData implements Comparable<BrokerData> {
    private String cluster;
    private String brokerName;
    private HashMap<Long/* brokerId */, String/* broker address */> brokerAddrs;

    private final Random random = new Random();
}

解析：

1. cluster：broker所属集群；
2. brokerName：broker组名称；
3. brokerAddrs：当前broker组包含的所有broker节点，key-brokerId（id=0代表master节点，id=其他 代表slave结点），value-broker网络地址；
4. random：随机种子，用于选择主节点时的随机种子（读写时，使用主节点，如果主接节点不存在，则在从节点列表中，随机选择一个）；

BrokerLiveInfo broker节点心跳、保活数据结构：

class BrokerLiveInfo {
    private long lastUpdateTimestamp;
    private DataVersion dataVersion;
    private Channel channel;
    private String haServerAddr;
}

解析：

1. lastUpdateTimestamp：上次心跳时间；
2. dataVersion：broker端上次数据版本；
3. channel：netty网络 长连接；
4. haServerAddr：组成 HA 高可用的同一组机器地址（brokerName相同）；

核心数据行为

获取集群信息

    public byte[] getAllClusterInfo() {
        ClusterInfo clusterInfoSerializeWrapper = new ClusterInfo();
        clusterInfoSerializeWrapper.setBrokerAddrTable(this.brokerAddrTable);
        clusterInfoSerializeWrapper.setClusterAddrTable(this.clusterAddrTable);
        return clusterInfoSerializeWrapper.encode();
    }

解析：封装并返回了 broker组-broker节点table、broker集群-broker组关系；

删除topic

    public void deleteTopic(final String topic) {
        try {
            try {
                this.lock.writeLock().lockInterruptibly();
                this.topicQueueTable.remove(topic);
            } finally {
                this.lock.writeLock().unlock();
            }
        } catch (Exception e) {
            log.error("deleteTopic Exception", e);
        }
    }

解析：获取写锁，删除topic-queuetable数据结构中的指定tpoic，释放写锁；

获取全部topic

    public byte[] getAllTopicList() {
        TopicList topicList = new TopicList();
        try {
            try {
                this.lock.readLock().lockInterruptibly();
                topicList.getTopicList().addAll(this.topicQueueTable.keySet());
            } finally {
                this.lock.readLock().unlock();
            }
        } catch (Exception e) {
            log.error("getAllTopicList Exception", e);
        }

        return topicList.encode();
    }

解析：获取写锁，获取topic-queue table表中的所有key，释放写锁；

注册broker节点

    public RegisterBrokerResult registerBroker(
        final String clusterName, //broker所在的集群名称
        final String brokerAddr, //broker地址
        final String brokerName, //broker名称
        final long brokerId, //brokerId
        final String haServerAddr, //broker的主备地址
        final TopicConfigSerializeWrapper topicConfigWrapper,   //broker上注册的topic信息
        final List<String> filterServerList, //过滤服务器列表
        final Channel channel) { //broker的连接通道
        RegisterBrokerResult result = new RegisterBrokerResult();
        try {
            try {
                // 获取写锁
                this.lock.writeLock().lockInterruptibly();

                // 获取所在集群名称对应的brokerName集合，如果不存在，则新建，如果存在，则更新
                Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
                if (null == brokerNames) {
                    brokerNames = new HashSet<String>();
                    this.clusterAddrTable.put(clusterName, brokerNames);
                }
                brokerNames.add(brokerName);

                // 是否是第一次注册
                boolean registerFirst = false;

                // 获取brokerName对应的BrokerData对象，如果不存在，则新建
                BrokerData brokerData = this.brokerAddrTable.get(brokerName);
                if (null == brokerData) {
                    registerFirst = true;
                    brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
                    this.brokerAddrTable.put(brokerName, brokerData);
                }
                Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
                //Switch slave to master: first remove <1, IP:PORT> in namesrv, then add <0, IP:PORT>
                //The same IP:PORT must only have one record in brokerAddrTable
                Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
                // 如果存在，则继续判断是否BorkerAddr相同，brokerId不同，如果不相同，则删除旧的BrokerAddr
                // 因为brokerAddrTable中存储的是brokerId和brokerAddr的映射关系
                while (it.hasNext()) {
                    Entry<Long, String> item = it.next();
                    if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
                        it.remove();
                    }
                }


                String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
                // 第一次注册条件：如果不存在，是第一次注册；如果存在，且brokerId、brokerAddr都不相同，则是第一次注册
                registerFirst = registerFirst || (null == oldAddr);

                // 如果topic配置不等于null，且brokerId为0，则更新topic配置
                if (null != topicConfigWrapper
                    && MixAll.MASTER_ID == brokerId) {
                    // 如果brokerTopic配置的版本号有变更 或者是第一次注册，则更新topic配置
                    if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                        || registerFirst) {
                        ConcurrentMap<String, TopicConfig> tcTable =
                            topicConfigWrapper.getTopicConfigTable();
                        if (tcTable != null) {
                            for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
                                this.createAndUpdateQueueData(brokerName, entry.getValue());
                            }
                        }
                    }
                }
                
                // 更新brokerLiveTable，将最近一次心跳时间，数据版本，长连接
                BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
                    new BrokerLiveInfo(
                        System.currentTimeMillis(),
                        topicConfigWrapper.getDataVersion(),
                        channel,
                        haServerAddr));
                if (null == prevBrokerLiveInfo) {
                    log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
                }
                
                // 更新filterServerTable
                if (filterServerList != null) {
                    if (filterServerList.isEmpty()) {
                        this.filterServerTable.remove(brokerAddr);
                    } else {
                        this.filterServerTable.put(brokerAddr, filterServerList);
                    }
                }
                
                // 如果brokerId不为0，则代表从节点，需要获取主节点的保活信息，封装主节点地址、ha集群地址返回给客户端
                if (MixAll.MASTER_ID != brokerId) {
                    String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
                    if (masterAddr != null) {
                        BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
                        if (brokerLiveInfo != null) {
                            result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
                            result.setMasterAddr(masterAddr);
                        }
                    }
                }
            } finally {
                this.lock.writeLock().unlock();
            }
        } catch (Exception e) {
            log.error("registerBroker Exception", e);
        }

        return result;
    }

解析：broker节点到namesrv注册topic配置信息，且维护了clusterAddrTable、brokerAddrTable、topicQueueTable、brokerLiveTable、filterServerTable配置信息，并封装返回了主节点地址、和HA高可用组地址。

权限处理

    public int wipeWritePermOfBrokerByLock(final String brokerName) {
        return operateWritePermOfBrokerByLock(brokerName, RequestCode.WIPE_WRITE_PERM_OF_BROKER);
    }

    public int addWritePermOfBrokerByLock(final String brokerName) {
        return operateWritePermOfBrokerByLock(brokerName, RequestCode.ADD_WRITE_PERM_OF_BROKER);
    }

    private int operateWritePermOfBrokerByLock(final String brokerName, final int requestCode) {
        try {
            try {
                this.lock.writeLock().lockInterruptibly();
                return operateWritePermOfBroker(brokerName, requestCode);
            } finally {
                this.lock.writeLock().unlock();
            }
        } catch (Exception e) {
            log.error("operateWritePermOfBrokerByLock Exception", e);
        }

        return 0;
    }


    private int operateWritePermOfBroker(final String brokerName, final int requestCode) {
        int topicCnt = 0;
        for (Entry<String, List<QueueData>> entry : this.topicQueueTable.entrySet()) {
            List<QueueData> qdList = entry.getValue();

            for (QueueData qd : qdList) {
                if (qd.getBrokerName().equals(brokerName)) {
                    int perm = qd.getPerm();
                    switch (requestCode) {
                        case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
                            perm &= ~PermName.PERM_WRITE;
                            break;
                        case RequestCode.ADD_WRITE_PERM_OF_BROKER:
                            perm = PermName.PERM_READ | PermName.PERM_WRITE;
                            break;
                    }
                    qd.setPerm(perm);
                    topicCnt++;
                }
            }
        }

        return topicCnt;
    }

取消注册broker节点

    public void unregisterBroker(
        final String clusterName,
        final String brokerAddr,
        final String brokerName,
        final long brokerId) {
        try {
            try {
                // 获取写锁
                this.lock.writeLock().lockInterruptibly();
                // 移除心跳、保活信息
                BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.remove(brokerAddr);
                log.info("unregisterBroker, remove from brokerLiveTable {}, {}",
                    brokerLiveInfo != null ? "OK" : "Failed",
                    brokerAddr
                );
                // 移除过滤器
                this.filterServerTable.remove(brokerAddr);

                // 移除brokerAddrTable的brokerName组
                boolean removeBrokerName = false;
                BrokerData brokerData = this.brokerAddrTable.get(brokerName);
                if (null != brokerData) {
                    String addr = brokerData.getBrokerAddrs().remove(brokerId);
                    log.info("unregisterBroker, remove addr from brokerAddrTable {}, {}",
                        addr != null ? "OK" : "Failed",
                        brokerAddr
                    );

                    if (brokerData.getBrokerAddrs().isEmpty()) {
                        this.brokerAddrTable.remove(brokerName);
                        log.info("unregisterBroker, remove name from brokerAddrTable OK, {}",
                            brokerName
                        );

                        removeBrokerName = true;
                    }
                }

                // 移除removeBrokerName后，removeBrokerNameSet为空，则移除clusterAddrTable中维护的brokerName
                if (removeBrokerName) {
                    Set<String> nameSet = this.clusterAddrTable.get(clusterName);
                    if (nameSet != null) {
                        boolean removed = nameSet.remove(brokerName);
                        log.info("unregisterBroker, remove name from clusterAddrTable {}, {}",
                            removed ? "OK" : "Failed",
                            brokerName);

                        if (nameSet.isEmpty()) {
                            this.clusterAddrTable.remove(clusterName);
                            log.info("unregisterBroker, remove cluster from clusterAddrTable {}",
                                clusterName
                            );
                        }
                    }
                    // 移除topicQueueTable中维护的brokerName
                    this.removeTopicByBrokerName(brokerName);
                }
            } finally {
                this.lock.writeLock().unlock();
            }
        } catch (Exception e) {
            log.error("unregisterBroker Exception", e);
        }
    }