1、启动入口
broker启动类在:src/main/java/org/apache/rocketmq/broker/BrokerStartup.java
public static void main(String[] args) {
start(createBrokerController(args));
}
可以看到mian()方法执行了两个操作:
1.createBrokerController():创建broker处理器
2.start():启动broker
分别分析一下这两个操作的内容:
2.createBrokerController()创建处理器
这个方法内容比较多,省略了一些命令行相关的代码
public static BrokerController createBrokerController(String[] args) {
//获取mq版本信息
......
try {
//解析命令行参数
......
// 实例化broker、客户端和服务端配置
final BrokerConfig brokerConfig = new BrokerConfig();
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
final NettyClientConfig nettyClientConfig = new NettyClientConfig();
// tls安全相关
nettyClientConfig.setUseTLS(Boolean.parseBoolean(System.getProperty(TLS_ENABLE,
String.valueOf(TlsSystemConfig.tlsMode == TlsMode.ENFORCING))));
// 配置端口
nettyServerConfig.setListenPort(10911);
//实例化消息存储配置
final MessageStoreConfig messageStoreConfig = new MessageStoreConfig();
//判断是否是从节点
if (BrokerRole.SLAVE == messageStoreConfig.getBrokerRole()) {
//从节点相关配置比例40-10
int ratio = messageStoreConfig.getAccessMessageInMemoryMaxRatio() - 10;
messageStoreConfig.setAccessMessageInMemoryMaxRatio(ratio);
}
......
// 将命令行中的配置设置到brokerConfig对象中
MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), brokerConfig);
// 检查环境变量:ROCKETMQ_HOME
if (null == brokerConfig.getRocketmqHome()) {
System.out.printf("Please set the %s variable in your environment to match the location of the RocketMQ installation", MixAll.ROCKETMQ_HOME_ENV);
System.exit(-2);
}
//获取连接信息
String namesrvAddr = brokerConfig.getNamesrvAddr();
//主从节点相关检查
switch (messageStoreConfig.getBrokerRole()) {
case ASYNC_MASTER:
case SYNC_MASTER:
brokerConfig.setBrokerId(MixAll.MASTER_ID);
break;
case SLAVE:
if (brokerConfig.getBrokerId() <= 0) {
System.out.printf("Slave's brokerId must be > 0");
System.exit(-3);
}
break;
default:
break;
}
......
// todo 实例化brokerController
final BrokerController controller = new BrokerController(
brokerConfig,
nettyServerConfig,
nettyClientConfig,
messageStoreConfig);
//注册配置
controller.getConfiguration().registerConfig(properties);
// todo 初始化
boolean initResult = controller.initialize();
if (!initResult) {
controller.shutdown();
System.exit(-3);
}
// 添加关闭钩子,在关闭前处理一些操作
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
private volatile boolean hasShutdown = false;
private AtomicInteger shutdownTimes = new AtomicInteger(0);
@Override
public void run() {
synchronized (this) {
log.info("Shutdown hook was invoked, {}", this.shutdownTimes.incrementAndGet());
if (!this.hasShutdown) {
this.hasShutdown = true;
long beginTime = System.currentTimeMillis();
controller.shutdown();
long consumingTimeTotal = System.currentTimeMillis() - beginTime;
log.info("Shutdown hook over, consuming total time(ms): {}", consumingTimeTotal);
}
}
}
}, "ShutdownHook"));
return controller;
} catch (Throwable e) {
e.printStackTrace();
System.exit(-1);
}
return null;
}
创建处理器时主要做了4个操作:
- 主从实相关的检查
- 实例化brokerController(包含实例化broker、客户端、服务端、消息存储等配置)
- 初始化controller.initialize()
- 添加关闭钩子,在关闭前处理一些操作
2.1实例化brokerController
demo中,BrokerConfig属性其实已经固定的或者是从命令行获取,在实际部署中可以通过配置conf下的文件定义部署模式,2m-2s-ASYNC 双主双从异步复制2m-2s-SYNC 双主双从同步双写等
brokerController的构造方法
public BrokerController(
final BrokerConfig brokerConfig,
final NettyServerConfig nettyServerConfig,
final NettyClientConfig nettyClientConfig,
final MessageStoreConfig messageStoreConfig
) {
// 4个核心配置信息
this.brokerConfig = brokerConfig;
this.nettyServerConfig = nettyServerConfig;
this.nettyClientConfig = nettyClientConfig;
this.messageStoreConfig = messageStoreConfig;
// 管理consumer消费消息的offset
this.consumerOffsetManager = new ConsumerOffsetManager(this);
// 管理topic配置
this.topicConfigManager = new TopicConfigManager(this);
// 处理 consumer 拉消息请求的
this.pullMessageProcessor = new PullMessageProcessor(this);
this.pullRequestHoldService = new PullRequestHoldService(this);
// 消息送达的监听器
this.messageArrivingListener
= new NotifyMessageArrivingListener(this.pullRequestHoldService);
...
// 往外发消息的组件
this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig);
...
}
构造方法定义了很多的对象,下面只列出其中一些作出说明:
- 核心配置赋值:主要是brokerConfig/nettyServerConfig/nettyClientConfig/messageStoreConfig四个配置
- ConsumerOffsetManager:管理consumer消费消息位置的偏移量,偏移量表示消费者组消费该topic消息的位置,后面再消费时,就从该位置后消费,避免重复消费消息,也避免了漏消费消息。
- topicConfigManager:topic配置管理器,管理如名称,topic队列数量等
- pullMessageProcessor:消息处理器,用来处理消费者拉消息
- messageArrivingListener:消息送达的监听器,当生产者的消息送达时,由该监听器监听
- brokerOuterAPI:往外发消息的组件,如向NameServer发送注册/注销消息
2.2初始化controller.initialize()
//BrokerController.class
public boolean initialize() throws CloneNotSupportedException {
// 加载......\store\config下的json文件数据,其中包含历史数据
boolean result = this.topicConfigManager.load();
result = result && this.consumerOffsetManager.load();
result = result && this.subscriptionGroupManager.load();
result = result && this.consumerFilterManager.load();
if (result) {
try {
// 消息存储管理组件
this.messageStore = new DefaultMessageStore(this.messageStoreConfig,
this.brokerStatsManager,
this.messageArrivingListener,
this.brokerConfig);
// 启用了DLeger,就创建DLeger相关组件
if (messageStoreConfig.isEnableDLegerCommitLog()) {
DLedgerRoleChangeHandler roleChangeHandler = new DLedgerRoleChangeHandler(this, (DefaultMessageStore) messageStore);
((DLedgerCommitLog)((DefaultMessageStore) messageStore).getCommitLog()).getdLedgerServer().getdLedgerLeaderElector().addRoleChangeHandler(roleChangeHandler);
}
// broker统计组件
this.brokerStats = new BrokerStats((DefaultMessageStore) this.messageStore);
//load plugin
MessageStorePluginContext context = new MessageStorePluginContext(messageStoreConfig, brokerStatsManager, messageArrivingListener, brokerConfig);
this.messageStore = MessageStoreFactory.build(context, this.messageStore);
this.messageStore.getDispatcherList().addFirst(new CommitLogDispatcherCalcBitMap(this.brokerConfig, this.consumerFilterManager));
} catch (IOException e) {
result = false;
log.error("Failed to initialize", e);
}
}
// 加载磁盘上的记录,如commitLog写入的位置、消费者主题/队列的信息
result = result && this.messageStore.load();
if (result) {
// 处理 nettyServer
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.clientHousekeepingService);
NettyServerConfig fastConfig = (NettyServerConfig) this.nettyServerConfig.clone();
fastConfig.setListenPort(nettyServerConfig.getListenPort() - 2);
this.fastRemotingServer = new NettyRemotingServer(fastConfig, this.clientHousekeepingService);
//诸多线程池的创建
......
//注册处理器
this.registerProcessor();
//诸多线程池的定义
......
}
初始化方法中大致的几个操作:
- 加载磁盘文件数据
- storePathRootDir\store\config下的json文件数据(consumerFilter.json,consumerOffset.json,subscriptionGroup.json,topics.json)
- storePathRootDir\store下的commitLog、Consume Queue、index等
- 定义broker的相关组件:消息存储管理组件、统计组件等
- 创建诸多线程池
- 注册处理器registerProcessor()
- 启动诸多定时任务
2.3注册处理器:BrokerController#registerProcessor
在实例化中,构造方法实际创建了很多不同的对象属性,比如像new SendMessageProcessor(this),this传入的是BrokerController实例,这些处理器对应的就是实例化创建对象属性的处理器
public void registerProcessor() {
/**
* SendMessageProcessor
*/
//定义发送消息处理器
SendMessageProcessor sendProcessor = new SendMessageProcessor(this);
sendProcessor.registerSendMessageHook(sendMessageHookList);
sendProcessor.registerConsumeMessageHook(consumeMessageHookList);
//分别向remotingServer和fastRemotingServer注册对应的处理器
this.remotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendProcessor, this.sendMessageExecutor);
......
this.fastRemotingServer.registerProcessor(RequestCode.SEND_MESSAGE, sendProcessor, this.sendMessageExecutor);
......
/**
* PullMessageProcessor
*/
......
/**
* ReplyMessageProcessor
*/
......
/**
* QueryMessageProcessor
*/
......
/**
* ClientManageProcessor
*/
......
/**
* ConsumerManageProcessor
*/
......
可以看到分别向remotingServer和fastRemotingServer注册对应的处理器,这俩个接口是干嘛的呢?先往下看,到启动broker的时候再说
2.4NettyRemotingService#registerProcessor
又回到了熟悉的NettyRemotingService,他主要是netty 服务端的一个操作对象,定义了start()启动方法,通信过程中主要以ByteBuf、RemotingCommand做为序列化对象,这里简单回顾一下继续往下看。
//NettyRemotingServer.class
class HandshakeHandler extends SimpleChannelInboundHandler<ByteBuf> {
}
class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
......
}
registerProcessor()方法会将这些处理器最终会保存在本地一个名为processorTable的HashMap集合中。
启动过后会在处理Netty服务交互时,执行发送Netty消息processRequestCommand()方法中把processorTable集合中对应的处理器拿出来调用
//NettyRemotingServer.class
@ChannelHandler.Sharable
class NettyServerHandler extends SimpleChannelInboundHandler<RemotingCommand> {
//netty提供的消息通道
@Override
protected void channelRead0(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
//调用消息抽象类的方法
processMessageReceived(ctx, msg);
}
}
}
//NettyRemotingAbstract.class
public void processMessageReceived(ChannelHandlerContext ctx, RemotingCommand msg) throws Exception {
final RemotingCommand cmd = msg;
if (cmd != null) {
//判断是请求还是响应状态
switch (cmd.getType()) {
case REQUEST_COMMAND:
//执行发送netty消息
processRequestCommand(ctx, cmd);
break;
case RESPONSE_COMMAND:
processResponseCommand(ctx, cmd);
break;
default:
break;
}
}
}
2.5添加关闭钩子
这一步和NameServer中差不多,在前面文章RocketMQ源码解析-NameServer 也介绍过就不做细述了。
3.broker启动start()
启动方法就是BrokerController.start(),broker中各个组件接口类都定义了启动和关闭的方法,启动broker时通过统一的start()入口去启动各个组件。
public void start() throws Exception {
//消息存储组件
if (this.messageStore != null) {
this.messageStore.start();
}
//消息交互组件,集成了Netty进行通信,作为服务端处理客户端的消息
if (this.remotingServer != null) {
this.remotingServer.start();
}
//消息交互组件,集成了Netty进行通信,作为服务端处理客户端的消息
if (this.fastRemotingServer != null) {
this.fastRemotingServer.start();
}
//启动文件监听器
if (this.fileWatchService != null) {
this.fileWatchService.start();
}
// 向 NameServer 服务端发送相关请求的连接与断开等,定时扫描ResponseTable并触发回调,也是一个netty服务
if (this.brokerOuterAPI != null) {
this.brokerOuterAPI.start();
}
//存储 pull Message 的请求, 并触发执行pull Message
if (this.pullRequestHoldService != null) {
this.pullRequestHoldService.start();
}
......
// broker 核心的心跳注册任务
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
if (this.brokerStatsManager != null) {
this.brokerStatsManager.start();
}
if (this.brokerFastFailure != null) {
this.brokerFastFailure.start();
}
}
3.1netty服务
可以看到启动了两个remotingServer,那么remotingServer和fastRemotingServer的区别是什么呢?
端口不同:
remotingServer:
- 监听listenPort配置项指定的监听端口,默认10911
fastRemotingServer:
- 监听端口值listenPort-2,即默认为10909
Broker端:
remotingServer:
- 可以处理客户端所有请求,如:生产者发送消息的请求,消费者拉取消息的请求。
fastRemotingServer :
- 功能基本与remotingServer相同,唯一不同的是没有注册 PullMessageProcessor . 也就是说 fastRemotingServer 不支持 pullMessage 请求不可以处理消费者拉取消息的请求。
- Broker在向NameServer注册时,只会上报remotingServer监听的listenPort端口。
3.2开启broker心跳注册
源码中创建了一个定时任务
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
//todo 处理消息注册的方法
BrokerController.this.registerBrokerAll(true, false, brokerConfig.isForceRegister());
} catch (Throwable e) {
log.error("registerBrokerAll Exception", e);
}
}
}, 1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)), TimeUnit.MILLISECONDS);
处理注册消息发送的时间间隔如下:
时间间隔可以自行配置,但不能小于10s,不能大于60s,默认是30s.
1000 * 10, Math.max(10000, Math.min(brokerConfig.getRegisterNameServerPeriod(), 60000)
接下来看消息注册的方法BrokerController.registerBrokerAll();
public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
//获取读取json文件后缓存到本地的topic信息
TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();
//topic相关配置处理
if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
|| !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
TopicConfig tmp =
new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
this.brokerConfig.getBrokerPermission());
topicConfigTable.put(topicConfig.getTopicName(), tmp);
}
topicConfigWrapper.setTopicConfigTable(topicConfigTable);
}
//todo 这里会判断是否需要进行注册,如果forceRegister=false, 则会执行needRegister()
if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
this.getBrokerAddr(),
this.brokerConfig.getBrokerName(),
this.brokerConfig.getBrokerId(),
this.brokerConfig.getRegisterBrokerTimeoutMills())) {
// todo 进行注册操作
doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
}
}
因为demo中forceRegister默认设置成true,如果为false就会调用needRegister()
private boolean needRegister(final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final int timeoutMills) {
TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();
// 像nameServer发送当前broker的信息,判断是否有变更需要重新进行注册
List<Boolean> changeList = brokerOuterAPI.needRegister(clusterName, brokerAddr, brokerName, brokerId, topicConfigWrapper, timeoutMills);
boolean needRegister = false;
// 发生了变化,就表示需要注册了返回true
for (Boolean changed : changeList) {
if (changed) {
needRegister = true;
break;
}
}
return needRegister;
}
上面就介绍了BrokerOuterAPI组件是向 NameServer 服务端发送相关请求,所以调用brokerOuterAPI.needRegister()就是进行上报broker信息
public List<Boolean> needRegister(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final TopicConfigSerializeWrapper topicConfigWrapper,
final int timeoutMills) {
final List<Boolean> changedList = new CopyOnWriteArrayList<>();
// 获取所有的 nameServer
List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
// 遍历所有的nameServer,逐一发送请求
for (final String namesrvAddr : nameServerAddressList) {
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
QueryDataVersionRequestHeader requestHeader
= new QueryDataVersionRequestHeader();
...
// 向nameServer发送消息,命令是 RequestCode.QUERY_DATA_VERSION
RemotingCommand request = RemotingCommand
.createRequestCommand(RequestCode.QUERY_DATA_VERSION, requestHeader);
// 把当前的 DataVersion 发到 nameServer
request.setBody(topicConfigWrapper.getDataVersion().encode());
// 发请求到nameServer
RemotingCommand response = remotingClient
.invokeSync(namesrvAddr, request, timeoutMills);
DataVersion nameServerDataVersion = null;
Boolean changed = false;
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
QueryDataVersionResponseHeader queryDataVersionResponseHeader =
(QueryDataVersionResponseHeader) response
.decodeCommandCustomHeader(QueryDataVersionResponseHeader.class);
changed = queryDataVersionResponseHeader.getChanged();
byte[] body = response.getBody();
if (body != null) {
// 拿到 DataVersion
nameServerDataVersion = DataVersion.decode(body, D
ataVersion.class);
// 这里是判断的关键
if (!topicConfigWrapper.getDataVersion()
.equals(nameServerDataVersion)) {
changed = true;
}
}
if (changed == null || changed) {
changedList.add(Boolean.TRUE);
}
}
default:
break;
}
...
} catch (Exception e) {
...
} finally {
countDownLatch.countDown();
}
}
});
}
try {
countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
log.error("query dataversion from nameserver countDownLatch await Exception", e);
}
}
return changedList;
}
- 这个方法里,先是遍历所有的nameServer,向每个nameServer都发送一条code为RequestCode.QUERY_DATA_VERSION的参数,参数为当前broker的DataVersion,DataVersion就是记录版本信息的对象。当nameServer收到消息后,就返回nameServer中保存的、与当前broker对应的DataVersion,当两者版本不相等时,就表明当前broker发生了变化,需要重新注册。
如果检测到broker信息发送了变化,那么就返回true调用doRegisterBrokerAll()执行注册操作
private void doRegisterBrokerAll(boolean checkOrderConfig, boolean oneway,
TopicConfigSerializeWrapper topicConfigWrapper) {
//装配注册列表、请求上下文,然后向nameServer集群中注册
List<RegisterBrokerResult> registerBrokerResultList = this.brokerOuterAPI.registerBrokerAll(
this.brokerConfig.getBrokerClusterName(),
this.getBrokerAddr(),
this.brokerConfig.getBrokerName(),
this.brokerConfig.getBrokerId(),
this.getHAServerAddr(),
topicConfigWrapper,
this.filterServerManager.buildNewFilterServerList(),
oneway,
this.brokerConfig.getRegisterBrokerTimeoutMills(),
this.brokerConfig.isCompressedRegister());
//拿到注册完borker的响应信息,后续进行本地同步更新处理
if (registerBrokerResultList.size() > 0) {
RegisterBrokerResult registerBrokerResult = registerBrokerResultList.get(0);
if (registerBrokerResult != null) {
if (this.updateMasterHAServerAddrPeriodically && registerBrokerResult.getHaServerAddr() != null) {
this.messageStore.updateHaMasterAddress(registerBrokerResult.getHaServerAddr());
}
this.slaveSynchronize.setMasterAddr(registerBrokerResult.getMasterAddr());
if (checkOrderConfig) {
this.getTopicConfigManager().updateOrderTopicConfig(registerBrokerResult.getKvTable());
}
}
}
}
brokerOuterAPI.registerBrokerAll()方法内也就是构建请求上下文,遍历nameServer列表,向netty服务端发送注册请求,接收到响应后装入registerBrokerResultList()集合中返回
public List<RegisterBrokerResult> registerBrokerAll(......) {
final List<RegisterBrokerResult> registerBrokerResultList = new CopyOnWriteArrayList<>();
//获取nameServer列表
List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
//构建请求报文
final RegisterBrokerRequestHeader requestHeader = new RegisterBrokerRequestHeader();
......
//遍历nameServer
for (final String namesrvAddr : nameServerAddressList) {
//通过专属线程池nameServer的netty服务端发送注册信息
brokerOuterExecutor.execute(new Runnable() {
@Override
public void run() {
try {
//执行注册
RegisterBrokerResult result = registerBroker(namesrvAddr,oneway, timeoutMills,requestHeader,body);
//装入list集合
if (result != null) {
registerBrokerResultList.add(result);
}
log.info("register broker[{}]to name server {} OK", brokerId, namesrvAddr);
} catch (Exception e) {
log.warn("registerBroker Exception, {}", namesrvAddr, e);
} finally {
countDownLatch.countDown();
}
}
});
}
.......
}
return registerBrokerResultList;
所谓的注册操作,就是当nameServer发送一条code为RequestCode.REGISTER_BROKER的消息,消息里会带上当前broker的topic信息、版本号等。
4.总结
核心的几个步骤:
- 首先创建BrokerController处理器:实例化加载各种配置后进行初始化
- 初始化:读取本地josn文件添加到缓存,向netty的远程接口类注册各种组件,以便在客户端和服务端交互时调用
- 添加关闭的钩子函数
- 执行borker启动:启动各种组件以及定义一个心跳的定时任务想NameServer注册broker信息,如果broekr发生变更在定时任务后续执行过程中也会向NameServer进行更新