作用
RocketMQ发送者Producer的作用是将消息发送到RocketMQ消息队列中。Producer负责将消息封装成消息对象，并将其发送到指定的主题（Topic）中。发送者可以根据需要选择同步发送或异步发送消息，并可以设置消息的延迟等特性。通过使用Producer，应用程序可以将消息发送到RocketMQ，以供其他消费者进行消费和处理。
源码
首先我们平常怎么使用RocketMQ发送消息的，下边时官网中的普通发送的示例代码。
public class SyncProducer {
public static void main(String[] args) throws Exception {
// 初始化一个producer并设置Producer group name
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name"); //（1）
// 设置NameServer地址
producer.setNamesrvAddr("localhost:9876"); //（2）
// 启动producer
producer.start();
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,`在这里插入代码片`
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body*/
// 利用producer进行发送，并同步等待发送结果
SendResult sendResult = producer.send(msg); //（4）
System.out.printf("%s%n", sendResult);
}
// 一旦producer不再使用，关闭producer
producer.shutdown();
}
}
可以看到启动producer的代码producer.start()，今天就学习producer的启动逻辑。入口时DefaultMQProducer.start(),最主要的逻辑时this.defaultMQProducerImpl.start()这段。我们继续向下边看
public void start(final boolean startFactory) throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
this.serviceState = ServiceState.START_FAILED;
//校验producerGroup
this.checkConfig();
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
this.defaultMQProducer.changeInstanceNameToPID();
}
//创建个MQClientInstance
this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQProducer, rpcHook);
//把当前的生产者实例注册到producerTable 生产者注册表中
boolean registerOK = mQClientFactory.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
//因为刚启动，所以tpic信息为空
this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
//启动生产者相关信息
if (startFactory) {
mQClientFactory.start();
}
log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
this.defaultMQProducer.isSendMessageWithVIPChannel());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The producer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
//发送心跳信息到所有broker
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
//删除超时请求，当请求超过设定时间还没有执行，则为超时请求，删除
this.timer.scheduleAtFixedRate(new TimerTask() {
@Override
public void run() {
try {
RequestFutureTable.scanExpiredRequest();
} catch (Throwable e) {
log.error("scan RequestFutureTable exception", e);
}
}
}, 1000 * 3, 1000);
}
这段逻辑注释中已解释清除，我们还是看主要逻辑：mQClientFactory.start()
public void start() throws MQClientException {
synchronized (this) {
switch (this.serviceState) {
case CREATE_JUST:
//先把服务器状态置为START_FAILED，防止重复启动
this.serviceState = ServiceState.START_FAILED;
// If not specified,looking address from name server
if (null == this.clientConfig.getNamesrvAddr()) {
//拉取nameServerAddr地址
this.mQClientAPIImpl.fetchNameServerAddr();
}
// Start request-response channel
this.mQClientAPIImpl.start();
// Start various schedule tasks
//各种定时任务，主要有：发送心跳信息，拉取nameServer地址信息，拉取topic信息
this.startScheduledTask();
// Start pull service 启动拉消息线程
this.pullMessageService.start();
// Start rebalance service，启动负载均衡线程，consumer服务数量发生改变，需要再平衡topic消费
this.rebalanceService.start();
// Start push service 再次调用生产者启动类，确保启动成功
this.defaultMQProducer.getDefaultMQProducerImpl().start(false);
log.info("the client factory [{}] start OK", this.clientId);
//服务状态设置为运行中
this.serviceState = ServiceState.RUNNING;
break;
case START_FAILED:
throw new MQClientException("The Factory object[" + this.getClientId() + "] has been created before, and failed.", null);
default:
break;
}
}
}
这里的逻辑也不算多，主要时启动各种线程和各种定时任务，我们挑两个逻辑仔细查看。一个是：拉取nameServer地址的；另一个是：启动各种定时任务的。这两个逻辑主要是讲topic路由信息的，后期消息发送需要这方面的知识。
nameServerAddr地址的拉取
nameSrvAddr拉取后是有一个和本地nameSrvAddr比较的，如果不同会更新本地nameSrvAddr。
public final String fetchNSAddr(boolean verbose, long timeoutMills) {
String url = this.wsAddr;
try {
if (!UtilAll.isBlank(this.unitName)) {
url = url + "-" + this.unitName + "?nofix=1";
}
//使用http请求获取nameSrvAddr地址
HttpTinyClient.HttpResult result = HttpTinyClient.httpGet(url, null, null, "UTF-8", timeoutMills);
if (200 == result.code) {
String responseStr = result.content;
if (responseStr != null) {
return clearNewLine(responseStr);
} else {
log.error("fetch nameserver address is null");
}
} else {
log.error("fetch nameserver address failed. statusCode=" + result.code);
}
} catch (IOException e) {
if (verbose) {
log.error("fetch name server address exception", e);
}
}
if (verbose) {
String errorMsg =
"connect to " + url + " failed, maybe the domain name " + MixAll.getWSAddr() + " not bind in /etc/hosts";
errorMsg += FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL);
log.warn(errorMsg);
}
return null;
}
上边这段就是拉取获取nameSrvAddr地址的逻辑，可以看到底层他是通过http方式获取nameSrvAddr的。那就有个疑问，这个url是从哪里来的呢？
看这段代码String url = this.wsAddr;继续向下走，可以看到这个wsAddr是对象TopAddressing的属性。找到有参构造代码的地方，可以看到在MQClientAPIImpl对象初始化的时候有对TopAddressing对象初始化。
public MQClientAPIImpl(final NettyClientConfig nettyClientConfig,
final ClientRemotingProcessor clientRemotingProcessor,
RPCHook rpcHook, final ClientConfig clientConfig) {
this.clientConfig = clientConfig;
topAddressing = new TopAddressing(MixAll.getWSAddr(), clientConfig.getUnitName());
查看getWSAddr()方法
public static String getWSAddr() {
String wsDomainName = System.getProperty("rocketmq.namesrv.domain", DEFAULT_NAMESRV_ADDR_LOOKUP);
String wsDomainSubgroup = System.getProperty("rocketmq.namesrv.domain.subgroup", "nsaddr");
String wsAddr = "http://" + wsDomainName + ":8080/rocketmq/" + wsDomainSubgroup;
if (wsDomainName.indexOf(":") > 0) {
wsAddr = "http://" + wsDomainName + "/rocketmq/" + wsDomainSubgroup;
}
return wsAddr;
}
可以看到这个nameSrvAdd地址是拿的系统变量。这个调用
- 到系统变量中拿到nameServer服务地址
- 根据地址通过http请求获取nameSrvAddr地址信息
- 和本地nameSrvAddr地址信息比较，不同则更新本地nameSrvAddr地址信息
定时任务
producer启动中的定时任务主要是这几个，拉取nameServer地址，拉取最新的topic路由信息，
发送心跳和更新filterServer，清除下线的broker，发送本地缓存的消费进度到broker，调整DefaultMQPushConsumer核心线程数。
其中获取nameServer地址的逻辑上边已经讲过了，逻辑是一样的。下面我们主要讲两个逻辑，一个是拉取topic路由消息，另一个是发送心跳逻辑
topic路由信息的更新
//延迟10ms，每30秒获取最新的topic信息，新老不同，更新本地的topic信息
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
MQClientInstance.this.updateTopicRouteInfoFromNameServer();
} catch (Exception e) {
log.error("ScheduledTask updateTopicRouteInfoFromNameServer exception", e);
}
}
}, 10, this.clientConfig.getPollNameServerInterval(), TimeUnit.MILLISECONDS);
延迟10ms，每30秒获取最新的topic信息，新老不同，更新本地的topic信息。进到代码里
//获取最新的topic路由信息，并
public void updateTopicRouteInfoFromNameServer() {
Set<String> topicList = new HashSet<String>();
// Consumer 遍历consumerTable中订阅关系的topic信息
{
Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQConsumerInner> entry = it.next();
MQConsumerInner impl = entry.getValue();
if (impl != null) {
Set<SubscriptionData> subList = impl.subscriptions();
if (subList != null) {
for (SubscriptionData subData : subList) {
topicList.add(subData.getTopic());
}
}
}
}
}
// Producer 遍历producerTable中已存在的topic信息
{
Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQProducerInner> entry = it.next();
MQProducerInner impl = entry.getValue();
if (impl != null) {
Set<String> lst = impl.getPublishTopicList();
topicList.addAll(lst);
}
}
}
//拿到本地所有topic信息去获取最新的topic信息
for (String topic : topicList) {
this.updateTopicRouteInfoFromNameServer(topic);
}
}
可以看到会遍历出本地所有的topic信息，其中包括consumerTable中订阅关系中的topic信息和producerTable中已存在的topic信息。拿到这些topic后会遍历topic获取最新的topic信息
接下里看this.updateTopicRouteInfoFromNameServer(topic)方法
public boolean updateTopicRouteInfoFromNameServer(final String topic, boolean isDefault,
DefaultMQProducer defaultMQProducer) {
try {
//加锁
if (this.lockNamesrv.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
try {
TopicRouteData topicRouteData;
if (isDefault && defaultMQProducer != null) {
topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),
1000 * 3);
if (topicRouteData != null) {
for (QueueData data : topicRouteData.getQueueDatas()) {
//设置topic队列数，可以看到读写队列设置数时一样的
int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
data.setReadQueueNums(queueNums);
data.setWriteQueueNums(queueNums);
}
}
} else {
//netty通信，查询topic路由信息
topicRouteData = this.mQClientAPIImpl.getTopicRouteInfoFromNameServer(topic, 1000 * 3);
}
if (topicRouteData != null) {
TopicRouteData old = this.topicRouteTable.get(topic);
boolean changed = topicRouteDataIsChange(old, topicRouteData);
if (!changed) {
changed = this.isNeedUpdateTopicRouteInfo(topic);
} else {
log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
}
//最新topic信息和本地信息不同，消息改变，更新本地信息
if (changed) {
TopicRouteData cloneTopicRouteData = topicRouteData.cloneTopicRouteData();
for (BrokerData bd : topicRouteData.getBrokerDatas()) {
this.brokerAddrTable.put(bd.getBrokerName(), bd.getBrokerAddrs());
}
// Update Pub info
{
TopicPublishInfo publishInfo = topicRouteData2TopicPublishInfo(topic, topicRouteData);
publishInfo.setHaveTopicRouterInfo(true);
Iterator<Entry<String, MQProducerInner>> it = this.producerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQProducerInner> entry = it.next();
MQProducerInner impl = entry.getValue();
if (impl != null) {
impl.updateTopicPublishInfo(topic, publishInfo);
}
}
}
// Update sub info
{
Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQConsumerInner> entry = it.next();
MQConsumerInner impl = entry.getValue();
if (impl != null) {
impl.updateTopicSubscribeInfo(topic, subscribeInfo);
}
}
}
log.info("topicRouteTable.put. Topic = {}, TopicRouteData[{}]", topic, cloneTopicRouteData);
this.topicRouteTable.put(topic, cloneTopicRouteData);
return true;
上边截取的是主要的逻辑，其中核心逻辑是通过netty网络去nameServer去拉取topic信息。这其中的逻辑有这么一段
for (QueueData data : topicRouteData.getQueueDatas()) {
//设置topic队列数，可以看到读写队列设置数时一样的
int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
data.setReadQueueNums(queueNums);
data.setWriteQueueNums(queueNums);
}
这段是设置topic队列数，可以看到读写队列设置数是一样的。总的流程是这样的
- 遍历出本地所有的topic信息，其中包括consumerTable中订阅关系中的topic信息和producerTable中已存在的topic信息
- 遍历本地topic信息，根据topic信息去nameServer拉取最新的topic信息
- 比较本地topic信息和最新的topic信息，不同的话更新本地topic信息为最新topic信息
发送心跳信息和清除下线的broker信息
//延迟1s，每30秒获取发送心跳信息和更新filterServer
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
try {
//清除下线的broker
MQClientInstance.this.cleanOfflineBroker();
//向所有broker发送心跳信息
MQClientInstance.this.sendHeartbeatToAllBrokerWithLock();
} catch (Exception e) {
log.error("ScheduledTask sendHeartbeatToAllBroker exception", e);
}
}
}, 1000, this.clientConfig.getHeartbeatBrokerInterval(), TimeUnit.MILLISECONDS);
这个定时任务不止是向所有broker发送心跳信息，还包括清除下线的broker和更新filterServer
我们首先看清除下线的broker，代码比较简单就不贴上来了，主要是做了一下逻辑
- 到注册表brokerAddrTable中拿到所有的broker地址信息
- 遍历注册表中的broker地址
- 最新的topicRouteTable注册表中的broker信息是否包含老的broker地址信息
- 如果不包含，删除brokerAddrTable中老的broker信息
发送心跳信息到所有broker
Iterator<Entry<String, HashMap<Long, String>>> it = this.brokerAddrTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, HashMap<Long, String>> entry = it.next();
String brokerName = entry.getKey();
HashMap<Long, String> oneTable = entry.getValue();
if (oneTable != null) {
for (Map.Entry<Long, String> entry1 : oneTable.entrySet()) {
Long id = entry1.getKey();
String addr = entry1.getValue();
if (addr != null) {
if (consumerEmpty) {
//只向master发送心跳
if (id != MixAll.MASTER_ID)
continue;
}
try {
int version = this.mQClientAPIImpl.sendHearbeat(addr, heartbeatData, 3000);
if (!this.brokerVersionTable.containsKey(brokerName)) {
this.brokerVersionTable.put(brokerName, new HashMap<String, Integer>(4));
}
this.brokerVersionTable.get(brokerName).put(addr, version);
if (times % 20 == 0) {
log.info("send heart beat to broker[{} {} {}] success", brokerName, id, addr);
log.info(heartbeatData.toString());
}
} catch (Exception e) {
上边是发送心跳的核心逻辑，发送心跳只向master发送心跳，
public int sendHearbeat(
final String addr,
final HeartbeatData heartbeatData,
final long timeoutMillis
) throws RemotingException, MQBrokerException, InterruptedException {
RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.HEART_BEAT, null);
request.setLanguage(clientConfig.getLanguage());
request.setBody(heartbeatData.encode());
//netty通信，向所有的broker的master发送心跳信息
RemotingCommand response = this.remotingClient.invokeSync(addr, request, timeoutMillis);
assert response != null;
switch (response.getCode()) {
case ResponseCode.SUCCESS: {
return response.getVersion();
}
default:
break;
}
throw new MQBrokerException(response.getCode(), response.getRemark(), addr);
}
可以看到是使用netty发送心跳信息到broker服务的。同时可以看到返回了Version版本信息，并放入到brokerVersionTable此注册表中。
总结
- producer的启动首先也是初始化加载一些信息，比如加载producerGroup，创建MQClientInstance，初始化topicPublishInfoTable注册表
- 启动的时候第二部就是拉取nameServer地址，因为后续的topic信息也是需要到nameServer上去拉取的
- 启动各种定时任务，包括拉取nameServer地址，topic路由信息的拉取，发送心跳信息等
- 同事也会启动PullMessageService线程，这里主要是拉取消息的逻辑，后期会相信学习
- 这里边也有一些细节点，比如：topic路由信息里读写队列数量是一样的，只给master发送心跳，各种定时任务的时间设置等，这些对我们平常排查问题还是有很大帮助的
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
