1.前言
前面几篇文章已经介绍了rocketmq的架构,如何使用以及生产上问题如何解决。接下来几篇文章将会介绍rocketmq的源码,我们在看问题的时候,会带着我们的问题去代码找答案,同时在代码中学习一些常见的代码技巧。接下来,我们先来探究一下NameSrv的源码。
2.源码分析
2.1 NameSrv加载过程
RocketMq的创建过程一般是,创建一个controller,然后启动这个controller。这两个步骤。
public static NamesrvController main0(String[] args) {
try {
//创建Namesrv的controller
NamesrvController controller = createNamesrvController(args);
// 启动controller
start(controller);
String tip = "The Name Server boot success. serializeType=" + RemotingCommand.getSerializeTypeConfigInThisServer();
log.info(tip);
System.out.printf("%s%n", tip);
return controller;
} catch (Throwable e) {
e.printStackTrace();
System.exit(-1);
}
return null;
}
2.1 创建NamesrvContoller
2.2.1 创建NamesrvContoller的基本流程
public static NamesrvController createNamesrvController(String[] args) throws IOException, JoranException {
//设置rocketmq.remoting.version这个环境变量的值
System.setProperty(RemotingCommand.REMOTING_VERSION_KEY, Integer.toString(MQVersion.CURRENT_VERSION));
//PackageConflictDetect.detectFastjson();
Options options = ServerUtil.buildCommandlineOptions(new Options());
//解析从启动类传建立的参数,并且封装成commandLine,-n表示设置namesrv的启动参数,-c设置配置文件的地址,-p表示打印配置参数
commandLine = ServerUtil.parseCmdLine("mqnamesrv", args, buildCommandlineOptions(options), new PosixParser());
if (null == commandLine) {
System.exit(-1);
return null;
}
//NamesrvConfig表示的是Namesrv的配置信息
final NamesrvConfig namesrvConfig = new NamesrvConfig();
//Netty的配置信息
final NettyServerConfig nettyServerConfig = new NettyServerConfig();
nettyServerConfig.setListenPort(9876);
//如果启动参数中含有-c,会读取-c后面跟的配置文件,并且赋值给namesrvConfig和nettyServerConfig
if (commandLine.hasOption('c')) {
String file = commandLine.getOptionValue('c');
if (file != null) {
InputStream in = new BufferedInputStream(new FileInputStream(file));
properties = new Properties();
properties.load(in);
MixAll.properties2Object(properties, namesrvConfig);
MixAll.properties2Object(properties, nettyServerConfig);
//并且将文件路径放到namesrvConfig中的configStorePath下面
namesrvConfig.setConfigStorePath(file);
System.out.printf("load config properties file OK, %s%n", file);
in.close();
}
}
//如果包含-p参数,便打印一下配置信息后退出
if (commandLine.hasOption('p')) {
InternalLogger console = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_CONSOLE_NAME);
MixAll.printObjectProperties(console, namesrvConfig);
MixAll.printObjectProperties(console, nettyServerConfig);
System.exit(0);
}
//解析命令行参数并加载到namesrvConfig中
MixAll.properties2Object(ServerUtil.commandLine2Properties(commandLine), namesrvConfig);
if (null == namesrvConfig.getRocketmqHome()) {
System.out.printf("Please set the %s variable in your environment to match the location of the RocketMQ installation%n", MixAll.ROCKETMQ_HOME_ENV);
System.exit(-2);
}
//初始化rocketmq的日志配置
LoggerContext lc = (LoggerContext) LoggerFactory.getILoggerFactory();
JoranConfigurator configurator = new JoranConfigurator();
configurator.setContext(lc);
lc.reset();
configurator.doConfigure(namesrvConfig.getRocketmqHome() + "/conf/logback_namesrv.xml");
log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
MixAll.printObjectProperties(log, namesrvConfig);
MixAll.printObjectProperties(log, nettyServerConfig);
//构建NamesrvCongfig
final NamesrvController controller = new NamesrvController(namesrvConfig, nettyServerConfig);
//将全局配置存储到controller的allConfigs这个properties
// remember all configs to prevent discard
controller.getConfiguration().registerConfig(properties);
return controller;
}
可以看出NamesrvController的构建主要分成两部分:
1.加载命令行参数中的配置信息,以及-c命令指定的文件配置信息,并且构造namesrvConfig和nettyServerConfig这两个配置类,并且将配置文件写入到controller的allconfigs这个properties中。
2.通过namesrvConfig和nettyserverConfig构造NamesrvController。
2.2.2 NamesrvContoller的基本组成
namesrv的基本构成如下:
public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig) {
//namesrv的基本配置类
this.namesrvConfig = namesrvConfig;
//namesrv远程通信的netty的配置类
this.nettyServerConfig = nettyServerConfig;
//namesrv用来管理kv配置的类
this.kvConfigManager = new KVConfigManager(this);
//namesrv管理topic、boker等注册信息类
this.routeInfoManager = new RouteInfoManager();
//namesrv监听broker下线或者上线的类
this.brokerHousekeepingService = new BrokerHousekeepingService(this);
//namesrv的配置类
this.configuration = new Configuration(
log,
this.namesrvConfig, this.nettyServerConfig
);
this.configuration.setStorePathFromConfig(this.namesrvConfig, "configStorePath");
}
接下来我们来依次查看namesrv每个核心组件的作用以及实现。
1. namesrv基础配置类 -- namesrvConfig
namesrvConfig是组成namesrv的基本配置类。
public class NamesrvConfig {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
//rocketmq.home.dir的目录
private String rocketmqHome = System.getProperty(MixAll.ROCKETMQ_HOME_PROPERTY, System.getenv(MixAll.ROCKETMQ_HOME_ENV));
//kvConfig是namesrv的配置真正存储的地方,是以json格式存储的,接下来我们将称其为kv配置文件
private String kvConfigPath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "kvConfig.json";
//namesrv配置文件持久化的地方
private String configStorePath = System.getProperty("user.home") + File.separator + "namesrv" + File.separator + "namesrv.properties";
private String productEnvName = "center";
private boolean clusterTest = false;
//是否开启顺序消息
private boolean orderMessageEnable = false;
}
2. namesrv网络配置类 -- nettyserverConfig
nettyserverConfig是namesrv中netty长连接使用的类,该类在NettyRemotingServer中已经介绍过。
public class NettyServerConfig implements Cloneable {
//namesrv监听端口号
private int listenPort = 8888;
//netty的worker线程数量
private int serverWorkerThreads = 8;
private int serverCallbackExecutorThreads = 0;
//netty中bossGroup数量
private int serverSelectorThreads = 3;
//最多同时运行发送多少个oneWay请求
private int serverOnewaySemaphoreValue = 256;
//最多同时发送多少个异步请求
private int serverAsyncSemaphoreValue = 64;
//netty长连接最大的空闲时间
private int serverChannelMaxIdleTimeSeconds = 120;
//netty发送方的bytebuffer的大小
private int serverSocketSndBufSize = NettySystemConfig.socketSndbufSize;
//netty接收方的bytebuffer的大小
private int serverSocketRcvBufSize = NettySystemConfig.socketRcvbufSize;
//netty的流量控制参数,当超过高水位时,channel将不可写,直到地域低水位时,channel才会变成可写状态
private int writeBufferHighWaterMark = NettySystemConfig.writeBufferHighWaterMark;
private int writeBufferLowWaterMark = NettySystemConfig.writeBufferLowWaterMark;
private int serverSocketBacklog = NettySystemConfig.socketBacklog;
//在netty中是否开启池化,开启池化后,当bytebuffer的数据被写完过后,不会立刻释放,会在下一次内存分配的时候再次使用,而达到内存重复利用
private boolean serverPooledByteBufAllocatorEnable = true;
}
3. namesrv的kv配置文件管理类 -- KVConfigManager
KVConfigManager是专门管理namesrv中的kv配置文件的。它对应的持久化文件是在namesrvConfig中的kvConfigPath里面,是一个json文件。如果是load方法会将其加载到内存中同时放入到configTable里面。
public class KVConfigManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
private final NamesrvController namesrvController;
private final ReadWriteLock lock = new ReentrantReadWriteLock();
private final HashMap<String/* Namespace */, HashMap<String/* Key */, String/* Value */>> configTable =
new HashMap<String, HashMap<String, String>>();
public KVConfigManager(NamesrvController namesrvController) {
this.namesrvController = namesrvController;
}
public void load() {
String content = null;
try {
//从磁盘中加载出配置文件,读取成一个json字符串
content = MixAll.file2String(this.namesrvController.getNamesrvConfig().getKvConfigPath());
} catch (IOException e) {
log.warn("Load KV config table exception", e);
}
if (content != null) {
//将json字符串反序列化到configTable中。
KVConfigSerializeWrapper kvConfigSerializeWrapper =
KVConfigSerializeWrapper.fromJson(content, KVConfigSerializeWrapper.class);
if (null != kvConfigSerializeWrapper) {
this.configTable.putAll(kvConfigSerializeWrapper.getConfigTable());
log.info("load KV config table OK");
}
}
}
//将对应namespace,中的key和value存储到configTable中,并且持久化到kv配置文件下面
public void putKVConfig(final String namespace, final String key, final String value) {
}
//通过读写锁解决并发问题
//并且将文件重新更新到kv配置文件中
public void persist() {
}
//删除kv配置文件
public void deleteKVConfig(final String namespace, final String key) {
}
//获取所有的kv配置文件内容
public byte[] getKVListByNamespace(final String namespace) {
}
//获取对应namespace下面的对应key的配置文件
public String getKVConfig(final String namespace, final String key) {
}
//打印所有的kv配置文件
public void printAllPeriodically() {
}
}
4.Namesrv的路由管理类 -- RouteInfoManager
RouteInfoManager是namesrv的核心组件,主要负责管理namesrv的路由信息。其实就将topic、broker等之间的映射关系以map的形式存储起来,方便检索。接下来我们看看routInfo的组成:
//注意:通过rocketmq的集群可以了解:
//分级:broker集群 - broker主从 - broker - messagequeue
//broker是可以由多个集群组成的。
//每个集群下面是多组主从broker节点,每对主从节点有相同的brokerName
//每组主从节点下面有多个broker,里面通过brokerId来标识每一个broker
//每个broker可能有多个topic,每个topic也可能存在多个broker主从中
public class RouteInfoManager {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
private final static long BROKER_CHANNEL_EXPIRED_TIME = 1000 * 60 * 2;
private final ReadWriteLock lock = new ReentrantReadWriteLock();
//topic的管理信息,topic属于哪个broker集群,一个broker集群下面有多少个messagequeue,由此可以看出topic是放在多个broker集群里面的
private final HashMap<String/* topic */, Map<String /* brokerName */ , QueueData>> topicQueueTable;
//broker主从 - broker关系
private final HashMap<String/* brokerName */, BrokerData> brokerAddrTable;
//集群 - broker主从 关系
private final HashMap<String/* clusterName */, Set<String/* brokerName */>> clusterAddrTable;
//broker - broker存活信息
private final HashMap<String/* brokerAddr */, BrokerLiveInfo> brokerLiveTable;
//broker - broker的过滤器地址,broker每次发送消息与收到消息的时候,可以先通过过滤器进行过滤,然后再传递给接收方
private final HashMap<String/* brokerAddr */, List<String>/* Filter Server */> filterServerTable;
}
在看看routeInfo提供的类,其实就是对其保存的集群信息,topic信息等进行增删改查。
删除某个topic:
//删除某个集群中的某个topic
public void deleteTopic(final String topic, final String clusterName) {}
将broker信息加入到routeInfo中
* 注册broker
* 1.注册进集群 - broker主从关系:clusterAddrTable
* 2.注册进broker主从 - broker关系:brokerAddrTable
* 3.如果同一个broker的brokerId发生改变,需要更新brokerAddrTable里面的地址关系
* 4.如果节点是主节点(因为只有主broker才能写数据),如果主broker的topic改变,需要创建并更新broker里面的topic信息
* 5.更新broker的保活信息
* 6.更新broker的filterServer信息
* 7.更新broker的HaServer信息,Haserver信息是放在master broker里面的
*/
public RegisterBrokerResult registerBroker(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId,
final String haServerAddr,
final TopicConfigSerializeWrapper topicConfigWrapper,
final List<String> filterServerList,
final Channel channel) {}
}
将broker的信息从routInfo中摘除掉:
public void unregisterBroker(
final String clusterName,
final String brokerAddr,
final String brokerName,
final long brokerId) {
}
根据topic返回具有该topic的所有broker的信息,比如broker的地址等
/**
* 根据topic返回具有该topic的所有broker的信息,比如broker的地址等
*
*/
public TopicRouteData pickupTopicRouteData(final String topic) {}
扫描boker里面超时时间超过120s的broker,并且将其移除掉(清除掉broker的fiterServer信息,topic的注册信息,broker集群的信息):
/**
* 扫描boker里面超时时间超过120s的broker,并且将其移除掉(清除掉broker的fiterServer信息,topic的注册信息,broker集群的信息),
* 同时关闭与broker的远程连接,所以broker与namesrv的保活时间是120s
*/
public int scanNotActiveBroker() {}
5.boker上线下线管理--BrokerHousekeepingService
BrokerHousekeepingService是一个ChannelEventListener的子类,前面我们将NettyRemotingServer的时候曾经讲过,NettyRemotingServer里面有一个NettyConnectManageHandler组件,他可以监听到channel的注册或者下线时间,并且会发送一个事件到nettyEventExecutor的阻塞队列中,nettyEventExecutor会启动一个线程去消费事件,并且根据事件调用注册的channelHandler的方法来执行处理逻辑。我们来看一看ChannelEventListener对于不同事件是如何处理的。
public class BrokerHousekeepingService implements ChannelEventListener {
private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.NAMESRV_LOGGER_NAME);
private final NamesrvController namesrvController;
public BrokerHousekeepingService(NamesrvController namesrvController) {
this.namesrvController = namesrvController;
}
@Override
public void onChannelConnect(String remoteAddr, Channel channel) {
}
@Override
public void onChannelClose(String remoteAddr, Channel channel) {
this.namesrvController.getRouteInfoManager().onChannelDestroy(remoteAddr, channel);
}
@Override
public void onChannelException(String remoteAddr, Channel channel) {
this.namesrvController.getRouteInfoManager().onChannelDestroy(remoteAddr, channel);
}
@Override
public void onChannelIdle(String remoteAddr, Channel channel) {
this.namesrvController.getRouteInfoManager().onChannelDestroy(remoteAddr, channel);
}
}
可以看出当channel销毁或者空闲(也就是broker与namesrv超过120s没有通信),便会调用routeInfManager中的onChannelDestroy,来清除掉broker在namesrv中的注册信息(这个操作在rountInfoManager组件中已经讲过)。
6.namesrv收到请求后如何处理--DefaultRequestProcessor
在将NettyRemotingServer中也说过,真正的处理请求的业务逻辑是在requestProcessor中的processRequest中实现的,所以我们来看一看DefaultRequestProcessor中的processRequest方法:
@Override
//DefaultRequestProcessor在收到请求后会调用processRequest方法来处理请求
public RemotingCommand processRequest(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
if (ctx != null) {
log.debug("receive request, {} {} {}",
request.getCode(),
RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
request);
}
//根据request的code判断出请求类型,调用不同的方法
switch (request.getCode()) {
case RequestCode.PUT_KV_CONFIG:
//比如100,便是将将请求加入到KVConfig中
return this.putKVConfig(ctx, request);
case RequestCode.GET_KV_CONFIG:
return this.getKVConfig(ctx, request);
case RequestCode.DELETE_KV_CONFIG:
return this.deleteKVConfig(ctx, request);
case RequestCode.QUERY_DATA_VERSION:
return queryBrokerTopicConfig(ctx, request);
case RequestCode.REGISTER_BROKER:
Version brokerVersion = MQVersion.value2Version(request.getVersion());
if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
return this.registerBrokerWithFilterServer(ctx, request);
} else {
return this.registerBroker(ctx, request);
}
case RequestCode.UNREGISTER_BROKER:
return this.unregisterBroker(ctx, request);
case RequestCode.GET_ROUTEINFO_BY_TOPIC:
return this.getRouteInfoByTopic(ctx, request);
case RequestCode.GET_BROKER_CLUSTER_INFO:
return this.getBrokerClusterInfo(ctx, request);
case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
return this.wipeWritePermOfBroker(ctx, request);
case RequestCode.ADD_WRITE_PERM_OF_BROKER:
return this.addWritePermOfBroker(ctx, request);
case RequestCode.GET_ALL_TOPIC_LIST_FROM_NAMESERVER:
return getAllTopicListFromNameserver(ctx, request);
case RequestCode.DELETE_TOPIC_IN_NAMESRV:
return deleteTopicInNamesrv(ctx, request);
case RequestCode.GET_KVLIST_BY_NAMESPACE:
return this.getKVListByNamespace(ctx, request);
case RequestCode.GET_TOPICS_BY_CLUSTER:
return this.getTopicsByCluster(ctx, request);
case RequestCode.GET_SYSTEM_TOPIC_LIST_FROM_NS:
return this.getSystemTopicListFromNs(ctx, request);
case RequestCode.GET_UNIT_TOPIC_LIST:
return this.getUnitTopicList(ctx, request);
case RequestCode.GET_HAS_UNIT_SUB_TOPIC_LIST:
return this.getHasUnitSubTopicList(ctx, request);
case RequestCode.GET_HAS_UNIT_SUB_UNUNIT_TOPIC_LIST:
return this.getHasUnitSubUnUnitTopicList(ctx, request);
case RequestCode.UPDATE_NAMESRV_CONFIG:
return this.updateConfig(ctx, request);
case RequestCode.GET_NAMESRV_CONFIG:
return this.getConfig(ctx, request);
default:
break;
}
return null;
}
可以看出,它其实是根据不同的requestCode来判断请求类型,然后调用不同的处理逻辑,我们这里以设置kvconfig为例:
public RemotingCommand putKVConfig(ChannelHandlerContext ctx,
RemotingCommand request) throws RemotingCommandException {
//构建空响应RemotingCommand
final RemotingCommand response = RemotingCommand.createResponseCommand(null);
//解析出请求头
final PutKVConfigRequestHeader requestHeader =
(PutKVConfigRequestHeader) request.decodeCommandCustomHeader(PutKVConfigRequestHeader.class);
//根据请求头拿到namespace和key
if (requestHeader.getNamespace() == null || requestHeader.getKey() == null) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("namespace or key is null");
return response;
}
//将namespace、key和value放入到kvcongfig中
this.namesrvController.getKvConfigManager().putKVConfig(
requestHeader.getNamespace(),
requestHeader.getKey(),
requestHeader.getValue()
);
//返回响应成功
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
它其实就是解析出对应namespace和key和value值,让后放入到kvconfig中。
2.3 NameSrv的启动流程
namesrv的启动主要是先初始化controller,然后在启动controller,即启动Namesrv中的线程池。
public static NamesrvController start(final NamesrvController controller) throws Exception {
if (null == controller) {
throw new IllegalArgumentException("NamesrvController is null");
}
//调用controller的初始化方法初始化NamesrvController
boolean initResult = controller.initialize();
if (!initResult) {
controller.shutdown();
System.exit(-3);
}
//注册JVM的钩子方法,这也是常见的写法,当jvm退出的时候,会回调这里方法,完成扫尾操作
Runtime.getRuntime().addShutdownHook(new ShutdownHookThread(log, (Callable<Void>) () -> {
//其实就是停掉Namesrv启动时的线程池
controller.shutdown();
return null;
}));
//启动controller
controller.start();
return controller;
}
我们先来看看初始化NamesrvController的初始化流程。
public boolean initialize() {
//从磁盘中读取namesrv的kv配置到内存中
this.kvConfigManager.load();
//构建NettyProcessor
this.remotingServer = new NettyRemotingServer(this.nettyServerConfig, this.brokerHousekeepingService);
this.remotingExecutor =
Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(), new ThreadFactoryImpl("RemotingExecutorThread_"));
//注册Processor
this.registerProcessor();
//启动定时任务,每10秒钟检查一次超过120秒没有像向namesrv发送心跳包的broker
this.scheduledExecutorService.scheduleAtFixedRate(NamesrvController.this.routeInfoManager::scanNotActiveBroker, 5, 10, TimeUnit.SECONDS);
//启动定时任务,每10分钟打印一次kvconfig中的内容
this.scheduledExecutorService.scheduleAtFixedRate(NamesrvController.this.kvConfigManager::printAllPeriodically, 1, 10, TimeUnit.MINUTES);
//如果开启了tls模式
if (TlsSystemConfig.tlsMode != TlsMode.DISABLED) {
// Register a listener to reload SslContext
try {
//构建一个FileWatchServic来监听tls的ca证书、key的文件,如果有被改动,便直接断开连接
fileWatchService = new FileWatchService(
new String[] {
TlsSystemConfig.tlsServerCertPath,
TlsSystemConfig.tlsServerKeyPath,
TlsSystemConfig.tlsServerTrustCertPath
},
new FileWatchService.Listener() {
boolean certChanged, keyChanged = false;
@Override
public void onChanged(String path) {
if (path.equals(TlsSystemConfig.tlsServerTrustCertPath)) {
log.info("The trust certificate changed, reload the ssl context");
//如果文件发生改变,重新加载ssl的key文件内容
reloadServerSslContext();
}
if (path.equals(TlsSystemConfig.tlsServerCertPath)) {
certChanged = true;
}
if (path.equals(TlsSystemConfig.tlsServerKeyPath)) {
keyChanged = true;
}
if (certChanged && keyChanged) {
log.info("The certificate and private key changed, reload the ssl context");
certChanged = keyChanged = false;
reloadServerSslContext();
}
}
private void reloadServerSslContext() {
((NettyRemotingServer) remotingServer).loadSslContext();
}
});
} catch (Exception e) {
log.warn("FileWatchService created error, can't load the certificate dynamically");
}
}
return true;
}
然后是启动namesrv
public void start() throws Exception {
//启动remotingServer
this.remotingServer.start();
if (this.fileWatchService != null) {
//启动文件监测系统
this.fileWatchService.start();
}
}
3.总结
1.NameSrv之间是相互通信的吗?
从上面源码分析,可以看出namesrv之间是没有通信的。
2.NameSrv是如何存储topic的路由信息的?
namesrv的路由信息是存储在routInfoManager的多个map里面的,比如brokerId和地址的关系等。
3.NameSrv是如何实现broker之间通信的?
namesrv是通过nettyRemotingServer与boker通信的,通过实现requestProcessor的preRequest方法,来处理不同逻辑,比如获取有哪些topic。
4.namesrv的是如何处理超过120分钟的broker的
broker会向namesrv发送心跳包,更新它的BrokerLiveInfo信息,并且namesrv会启动一个线程池,每过10秒钟去扫描一遍routeInfoManager里面的BrokerLiveInfo的信息,超过120秒没有发送心跳包,便将它删除掉。