本系列参考书籍：《RocketMQ技术内幕》下载源码：https://github.com/apache/rocketmq.git
一、从路由中心谈起
首先来张图，看看RocketMQ到底是在干嘛。
说白了，就是接收生产者的消息，然后传递给消费者。虽然以前没有用过metaQ，但是这样一看又感觉到万物一理了，有空考虑将以前用到开源ROS的消息模式进行分析比较，看看有啥个区别和优劣点。
从上图我们也能发现一个问题，就是rocketMQ是基于Group机制的发布订阅模式，所以metaQ本质上就支持消息负载均衡。比如某个Topic有9条消息，其中一个Consumer Group有3个实例（3个进程 OR 3台机器），那么每个实例将均摊消费3条消息。
再来看个rocketMQ的物理部署图：
从图中我们可以看到，相比较于kafka的zookeeper大脑，rocketMQ的大脑是NameServer，Broker消息服务器（消息存储）、producter集群、consumer集群需要在NameServer进行注册，然后才能进行通信。
NameServer的启动流程看下org.apache.rocketmq.namesrv.NamesrvStartup，这个地方感觉没啥，就是创建了业务参数和网络参数。需要注意的是，在这里有个所谓的心跳检测，其实就是看看Broker的状态有没有挂掉，挂掉了就执行相应的逻辑，这个在下面会进行源码上的分析。
NameServer路由注册和剔除
前面说了，NameServer在监听着Broker的状态，那么先来看看Broker的哪些数据需要给它，路由注册和剔除就是围绕着这些数据进行的，路由注册和删除就是注册和删除这个数据，当然，这一大堆数据有个名词，叫做路由元消息。（以下的源码来自于org.apache.rocketmq.namesrv.routeinfo.RouteInfoManager）
{
// 消息队列路由消息
private final HashMap<String/* topic */, List<QueueData>> topicQueueTable;
// Broker基础消息
private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
// Broker集群消息
private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
// Broker状态消息
private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
// Broker上的filterServer列表，意思就是Broker把消息过滤后才发送给消费者
private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}理一下逻辑，NameServer集群中有很多NameServer，NameServer集群管着Broker集群，其中的Broker是Master-Slave架构，每个Broker中存着Topic，一个Topic又有着多个消息队列。
路由注册
意思就是我要把Broker注册到NameServer上面去，首先Broker启动就向NameServer发送心跳语句，每隔30s还向NameServer发送心跳包，然后NameServer就根据收到的心跳包更新它存储的路由元消息。
发送心跳包的代码就不展开了，就是设置了一个定时任务，向NameServer集群中的每个NameServer发送心跳包，网络传输基于Netty，这里需要看一下心跳包里具体的东西，后面才知道我们要处理哪些东西。
{
// broker地址
requestHeader.setBrokerAddr(brokerAddr);
// 判断是master还是slave；0代表master
requestHeader.setBrokerId(brokerId);
// broker名称
requestHeader.setBrokerName(brokerName);
// 集群名称
requestHeader.setClusterName(clusterName);
// master地址
requestHeader.setHaServerAddr(haServerAddr);
//
requestHeader.setCompressed(compressed);
// 请求体
RegisterBrokerBody requestBody = new RegisterBrokerBody();
RegisterBrokerBody requestBody = new RegisterBrokerBody();
// 主题配置，包含的信息；topicConfigTable
requestBody.setTopicConfigSerializeWrapper(topicConfigWrapper);
requestBody.setFilterServerList(filterServerList);
}NameServer处理心跳包，我们看下源码：
public RegisterBrokerResult registerBroker(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final String haServerAddr,
final TopicConfigSerializeWrapper topicConfigWrapper,
final List<String> filterServerList,
final Channel channel) {
RegisterBrokerResult result = new RegisterBrokerResult();
try {
try {
// 加写锁
this.lock.writeLock().lockInterruptibly();
// 1.根据集群名称判断判断当下是不是已经存在了broker集群。如没有，创建broker集群
Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
if (null == brokerNames) {
brokerNames = new HashSet<String>();
this.clusterAddrTable.put(clusterName, brokerNames);
}
brokerNames.add(brokerName);
// 是不是第一次注册
boolean registerFirst = false;
// 2.获取broker基础信息
BrokerData brokerData = this.brokerAddrTable.get(brokerName);
// 2.1如果为空的话进行组装并加入到brokerAddrTable中
if (null == brokerData) {
registerFirst = true;
brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
this.brokerAddrTable.put(brokerName, brokerData);
}
/* BrokerData所包含的属性值
* private String cluster;
* private String brokerName;
* //在这里brokerId=0 代表是Master
* private HashMap<Long/* brokerId */, String/* broker address */> brokerAddrs;
*/
Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
//Switch slave to master: first remove <1, IP:PORT> in namesrv, then add <0, IP:PORT>
//The same IP:PORT must only have one record in brokerAddrTable
// 注意此处是根据传入的brokerId判断是不是需要将此broker进行切换Master
Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
while (it.hasNext()) {
Entry<Long, String> item = it.next();
if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
it.remove();
}
}
// 2.2如果存在的话替换原来的
String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
registerFirst = registerFirst || (null == oldAddr);
// 3.填充topicQueueTable
if (null != topicConfigWrapper
&& MixAll.MASTER_ID == brokerId) {
if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
|| registerFirst) {
ConcurrentMap<String, TopicConfig> tcTable =
topicConfigWrapper.getTopicConfigTable();
if (tcTable != null) {
for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
// 可以进入到createAndUpdateQueueData函数去看创建QueueData的逻辑
this.createAndUpdateQueueData(brokerName, entry.getValue());
}
}
}
}
// 4.更新存活broker列表
BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
new BrokerLiveInfo(
System.currentTimeMillis(),
topicConfigWrapper.getDataVersion(),
channel,
haServerAddr));
if (null == prevBrokerLiveInfo) {
log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
}
// 5.注册过滤器列表
if (filterServerList != null) {
if (filterServerList.isEmpty()) {
this.filterServerTable.remove(brokerAddr);
} else {
this.filterServerTable.put(brokerAddr, filterServerList);
}
}
if (MixAll.MASTER_ID != brokerId) {
String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
if (masterAddr != null) {
BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
if (brokerLiveInfo != null) {
result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
result.setMasterAddr(masterAddr);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("registerBroker Exception", e);
}
return result;
}注：代码中使用的读写锁是并发读的，但是处理心跳包是串行的。
路由删除
路由删除的条件：
1.NameServer每10s扫描brokerLiveTable（HashMap<String, /* brokerAddr */BrokerLiveInfo>(256);），检测每个broker上次的心跳包与当前系统时间的差值，大于120s则移除broker；
2.broker关闭不正常。
其实删除的逻辑和增加的逻辑差不多是反过来的，根据前面NameServe中存储的路由元信息就可以知道大致我们要干嘛。首先根据remoteAddr从brokerLiveTable,filterServerTable移除broker的相关信息，然后维护下broker基础信息，集群信息，topic信息即可。在这里，我们的出发点是remoteAddr。
public void onChannelDestroy(String remoteAddr, Channel channel) {
String brokerAddrFound = null;
if (channel != null) {
try {
try {
//1 。申请读锁，看看broker是不是存在
this.lock.readLock().lockInterruptibly();
Iterator<Entry<String, BrokerLiveInfo>> itBrokerLiveTable =
this.brokerLiveTable.entrySet().iterator();
while (itBrokerLiveTable.hasNext()) {
Entry<String, BrokerLiveInfo> entry = itBrokerLiveTable.next();
if (entry.getValue().getChannel() == channel) {
brokerAddrFound = entry.getKey();
break;
}
}
} finally {
this.lock.readLock().unlock();
}
} catch (Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
if (null == brokerAddrFound) {
brokerAddrFound = remoteAddr;
} else {
log.info("the broker's channel destroyed, {}, clean it's data structure at once", brokerAddrFound);
}
if (brokerAddrFound != null && brokerAddrFound.length() > 0) {
try {
try {
// 2.申请写锁，开始移除。。
this.lock.writeLock().lockInterruptibly();
// 3.从brokerLiveTable和filterServerTable中移除
this.brokerLiveTable.remove(brokerAddrFound);
this.filterServerTable.remove(brokerAddrFound);
String brokerNameFound = null;
boolean removeBrokerName = false;
// 4.维护BrokerAddrTable
Iterator<Entry<String, BrokerData>> itBrokerAddrTable =
this.brokerAddrTable.entrySet().iterator();
while (itBrokerAddrTable.hasNext() && (null == brokerNameFound)) {
BrokerData brokerData = itBrokerAddrTable.next().getValue();
Iterator<Entry<Long, String>> it = brokerData.getBrokerAddrs().entrySet().iterator();
while (it.hasNext()) {
Entry<Long, String> entry = it.next();
Long brokerId = entry.getKey();
String brokerAddr = entry.getValue();
if (brokerAddr.equals(brokerAddrFound)) {
brokerNameFound = brokerData.getBrokerName();
it.remove();
log.info("remove brokerAddr[{}, {}] from brokerAddrTable, because channel destroyed",
brokerId, brokerAddr);
break;
}
}
if (brokerData.getBrokerAddrs().isEmpty()) {
removeBrokerName = true;
itBrokerAddrTable.remove();
log.info("remove brokerName[{}] from brokerAddrTable, because channel destroyed",
brokerData.getBrokerName());
}
}
// 5.从cluser集群中移除broker，然后看看是不是只有一个broker，是的话把集群也移除
if (brokerNameFound != null && removeBrokerName) {
Iterator<Entry<String, Set<String>>> it = this.clusterAddrTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, Set<String>> entry = it.next();
String clusterName = entry.getKey();
Set<String> brokerNames = entry.getValue();
boolean removed = brokerNames.remove(brokerNameFound);
if (removed) {
log.info("remove brokerName[{}], clusterName[{}] from clusterAddrTable, because channel destroyed",
brokerNameFound, clusterName);
if (brokerNames.isEmpty()) {
log.info("remove the clusterName[{}] from clusterAddrTable, because channel destroyed and no broker in this cluster",
clusterName);
it.remove();
}
break;
}
}
}
// 6.移除包含当前broker的队列，如果topic中只有当前broker的队列，将topic也移除
if (removeBrokerName) {
Iterator<Entry<String, List<QueueData>>> itTopicQueueTable =
this.topicQueueTable.entrySet().iterator();
while (itTopicQueueTable.hasNext()) {
Entry<String, List<QueueData>> entry = itTopicQueueTable.next();
String topic = entry.getKey();
List<QueueData> queueDataList = entry.getValue();
Iterator<QueueData> itQueueData = queueDataList.iterator();
while (itQueueData.hasNext()) {
QueueData queueData = itQueueData.next();
if (queueData.getBrokerName().equals(brokerNameFound)) {
itQueueData.remove();
log.info("remove topic[{} {}], from topicQueueTable, because channel destroyed",
topic, queueData);
}
}
if (queueDataList.isEmpty()) {
itTopicQueueTable.remove();
log.info("remove topic[{}] all queue, from topicQueueTable, because channel destroyed",
topic);
}
}
}
} finally {
this.lock.writeLock().unlock();
}
} catch (Exception e) {
log.error("onChannelDestroy Exception", e);
}
}
}操作topic是根据brokerName进行的，QueueData含有的属性：
private String brokerName;
private int readQueueNums;
private int writeQueueNums;
private int perm;
private int topicSynFlag;
疑问：等找到答案再来解答。。
- 话说NameServer也是集群，读源码的时候咋就没感觉到集群的存在意义，
- 在registerBroker函数中的第三步是要进行master判断才进行创建topic元数据？？
- 路由删除的channel是干嘛的，点进源码看是netty通道？
1813
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
