为什么官方建议：
自动创建topic机制，建议线下开启，线上关闭。
rocketmq在发送消息时，会先去获取topic的路由信息，如果topic是第一次发送消息，由于nameserver没有topic的路由信息，所以会再次以“TBW102”这个默认topic获取路由信息，假设broker都开启了自动创建开关，那么此时会获取所有broker的路由信息，消息的发送会根据负载算法选择其中一台Broker发送消息，消息到达broker后，发现本地没有该topic，会在创建该topic的信息塞进本地缓存中，同时会将topic路由信息注册到nameserver中，那么这样就会造成一个后果：以后所有该topic的消息，都将发送到这台broker上，如果该topic消息量非常大，会造成某个broker上负载过大，这样的消息存储就达不到负载均衡的效果了。
————————————————
版权声明：本文为CSDN博主「后端进阶」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/zchdjb/article/details/90795265
1、producer 生产消息
producer在发送消息之前，会先去nameSrv拉取topic信息(TopicPublishInfo)，
TopicPublishInfo
MessageQueue
TopicRouteData
QueueData
BrokerData
以上数据结构的分析如下：
QueueData,一个逻辑上的概念，代表了一个topic的所拥有的queue信息，
比如在broker-A上有2个读队列，2个写队列
MessageQueue就是实际的队列了，
一个QueueData相当于描述了多个MessageQueue，取决于QueueData属性readQueueNums,writeQueueNums
参考
调用tryToFindTopicPublishInfo()获取TopicPublishInfo
private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) {
// 先尝试从本地获取topic信息
TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
// 本地获取为空，或者messageQueue为空
if (null == topicPublishInfo || !topicPublishInfo.ok()) {
// 保存一个空的topicPublishInfo
this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
// 从namesrv拉取topic信息，包含broker的地址
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
// 重新复制topicPublishInfo
topicPublishInfo = this.topicPublishInfoTable.get(topic);
}
if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
// topicPublishInfo 可用时，直接返回
return topicPublishInfo;
} else {
// topicPublishInfo不可用
// 更新topic信息，但是传递了isDefault=true
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
// 返回的是topic的真实信息，或者是默认topic(TBW102)的配置信息（修改了queueNums）
topicPublishInfo = this.topicPublishInfoTable.get(topic);
return topicPublishInfo;
}
}（1）先从本地获取topicPublishInfo,
（2）本地获取不到时，尝试从nameSrv获取
（3）如果从nameSrv获取到时，直接返回，获取不到时，会再次调用updateTopicRouteInfoFFromNameServer(), 但是传递了isDefault=true,
会走updateTopicRouteInfoFromNameServer()中的代码如下：
if (isDefault && defaultMQProducer != null) {
// 获取默认topic=TBW102的路由信息
topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
1000 * 3);
if (topicRouteData != null) {
for (QueueData data : topicRouteData.getQueueDatas()) {
// defaultTopicQueueNums = 4;
// data.getReadQueueNums = 8
// 取较小的一个，然后每个queueData都重新赋值
int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
data.setReadQueueNums(queueNums);
data.setWriteQueueNums(queueNums);
}
}
}会从nameSrv中获取默认自动创建的TBW102的topicRoute信息，然后可能会修改readQueueNums和writeQueueNums数量
然后会将topicRuteData封装成TopicPublishInfo，保存在变量producerTable中，
然后在外层topicPublishInfoTable中，就可以获取到topic对应的topicPublishInfo了，
然后在获取到的topicPublishInfo中，先选取一个broker，再在broker上选取一个messageQueue,随后将message发送到这个选中的messageQueue上。
注意：上面获取的topicRoute信息，里面的broker信息，可能不是集群的所有broker信息，虽然是默认TBW102的所有broker信息，但是只有开启了自动创建topic的broker，在初始化的时候，才会去初始化默认TBW102的topic信息。
BrokerStartup.createBrokerController()-->new BrokerController()--> new TopicConfigManager()
在TopicConfigManager的构造函数中
{
// broker配置了autoCreateTopicEnable=true
// 自动创建topic
if (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
String topic = TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
TopicValidator.addSystemTopic(topic);
// 读队列，默认8个
topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
// 写队列，默认8个
topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
// 读写信息
int perm = PermName.PERM_INHERIT | PermName.PERM_READ | PermName.PERM_WRITE;
topicConfig.setPerm(perm);
// 将自动创建的topic信息保存到topicConfigTable
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
}2. broker处理broker发送的消息
前面分析过，producer生产消息，发送的消息类型是SEND_MESSAGE_V2，broker服务上，经由sendMessageProcessor来处理
经由asyncSendMessage()方法来处理，回先调用preSend()方法
private RemotingCommand preSend(ChannelHandlerContext ctx, RemotingCommand request,
SendMessageRequestHeader requestHeader) {
final RemotingCommand response = RemotingCommand.createResponseCommand(SendMessageResponseHeader.class);
response.setOpaque(request.getOpaque());
response.addExtField(MessageConst.PROPERTY_MSG_REGION, this.brokerController.getBrokerConfig().getRegionId());
response.addExtField(MessageConst.PROPERTY_TRACE_SWITCH, String.valueOf(this.brokerController.getBrokerConfig().isTraceOn()));
log.debug("Receive SendMessage request command {}", request);
final long startTimestamp = this.brokerController.getBrokerConfig().getStartAcceptSendRequestTimeStamp();
if (this.brokerController.getMessageStore().now() < startTimestamp) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(String.format("broker unable to service, until %s", UtilAll.timeMillisToHumanString2(startTimestamp)));
return response;
}
response.setCode(-1);
// 调用父类的msgCheck()方法
super.msgCheck(ctx, requestHeader, response);
if (response.getCode() != -1) {
return response;
}
return response;
}--
protected RemotingCommand msgCheck(final ChannelHandlerContext ctx,
final SendMessageRequestHeader requestHeader, final RemotingCommand response) {
if (!PermName.isWriteable(this.brokerController.getBrokerConfig().getBrokerPermission())
&& this.brokerController.getTopicConfigManager().isOrderTopic(requestHeader.getTopic())) {
response.setCode(ResponseCode.NO_PERMISSION);
response.setRemark("the broker[" + this.brokerController.getBrokerConfig().getBrokerIP1()
+ "] sending message is forbidden");
return response;
}
if (!TopicValidator.validateTopic(requestHeader.getTopic(), response)) {
return response;
}
if (TopicValidator.isNotAllowedSendTopic(requestHeader.getTopic(), response)) {
return response;
}
TopicConfig topicConfig =
this.brokerController.getTopicConfigManager().selectTopicConfig(requestHeader.getTopic());
if (null == topicConfig) {
int topicSysFlag = 0;
if (requestHeader.isUnitMode()) {
if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
} else {
topicSysFlag = TopicSysFlag.buildSysFlag(true, false);
}
}
log.warn("the topic {} not exist, producer: {}", requestHeader.getTopic(), ctx.channel().remoteAddress());
// 调用创建topic方法
topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageMethod(
requestHeader.getTopic(),
requestHeader.getDefaultTopic(),
RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
requestHeader.getDefaultTopicQueueNums(), topicSysFlag);
//
if (null == topicConfig) {
if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
topicConfig =
this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
requestHeader.getTopic(), 1, PermName.PERM_WRITE | PermName.PERM_READ,
topicSysFlag);
}
}
if (null == topicConfig) {
response.setCode(ResponseCode.TOPIC_NOT_EXIST);
response.setRemark("topic[" + requestHeader.getTopic() + "] not exist, apply first please!"
+ FAQUrl.suggestTodo(FAQUrl.APPLY_TOPIC_URL));
return response;
}
}
int queueIdInt = requestHeader.getQueueId();
int idValid = Math.max(topicConfig.getWriteQueueNums(), topicConfig.getReadQueueNums());
if (queueIdInt >= idValid) {
String errorInfo = String.format("request queueId[%d] is illegal, %s Producer: %s",
queueIdInt,
topicConfig.toString(),
RemotingHelper.parseChannelRemoteAddr(ctx.channel()));
log.warn(errorInfo);
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark(errorInfo);
return response;
}
return response;
}--
public TopicConfig createTopicInSendMessageMethod(final String topic, final String defaultTopic,
final String remoteAddress, final int clientDefaultTopicQueueNums, final int topicSysFlag) {
TopicConfig topicConfig = null;
boolean createNew = false;
try {
if (this.topicConfigTableLock.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
topicConfig = this.topicConfigTable.get(topic);
// 获取到topicConfig不为空，直接返回
if (topicConfig != null)
return topicConfig;
//获取不到topicConfig
// 获取默认topic的topicConfig
TopicConfig defaultTopicConfig = this.topicConfigTable.get(defaultTopic);
if (defaultTopicConfig != null) {
// 判断默认topic是TBW102
if (defaultTopic.equals(TopicValidator.AUTO_CREATE_TOPIC_KEY_TOPIC)) {
// broker配置autoCreateTopicEnable=false
// 就设置读写模式为可读可写
if (!this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
defaultTopicConfig.setPerm(PermName.PERM_READ | PermName.PERM_WRITE);
}
}
// 默认topic读写模式为继承
if (PermName.isInherited(defaultTopicConfig.getPerm())) {
topicConfig = new TopicConfig(topic);
int queueNums = Math.min(clientDefaultTopicQueueNums, defaultTopicConfig.getWriteQueueNums());
if (queueNums < 0) {
queueNums = 0;
}
topicConfig.setReadQueueNums(queueNums);
topicConfig.setWriteQueueNums(queueNums);
int perm = defaultTopicConfig.getPerm();
perm &= ~PermName.PERM_INHERIT;
topicConfig.setPerm(perm);
topicConfig.setTopicSysFlag(topicSysFlag);
topicConfig.setTopicFilterType(defaultTopicConfig.getTopicFilterType());
} else {
log.warn("Create new topic failed, because the default topic[{}] has no perm [{}] producer:[{}]",
defaultTopic, defaultTopicConfig.getPerm(), remoteAddress);
}
} else {
log.warn("Create new topic failed, because the default topic[{}] not exist. producer:[{}]",
defaultTopic, remoteAddress);
}
if (topicConfig != null) {
log.info("Create new topic by default topic:[{}] config:[{}] producer:[{}]",
defaultTopic, topicConfig, remoteAddress);
// 将topicconfig保存在本地缓存中
this.topicConfigTable.put(topic, topicConfig);
this.dataVersion.nextVersion();
createNew = true;
this.persist();
}
} finally {
this.topicConfigTableLock.unlock();
}
}
} catch (InterruptedException e) {
log.error("createTopicInSendMessageMethod exception", e);
}
if (createNew) {
// 新创建的topic，立即上报broker信息，将topic同步给namesrv
this.brokerController.registerBrokerAll(false, true, true);
}
return topicConfig;
}之后会将topicconfig保存在本地缓存中，如果是新创建的topic，就立即向nameSrv发送一下broker信息，到这里，topic已经创建完成
7947
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
