2.1 同步发送
同步发送是最常用的方式,是指消息发送方发出一条消息后,会在收到服务端同步响应之后才发下一条消息的通讯方式,可靠的同步传输被广泛应用于各种场景,如重要的通知消息、短消息通知等。
先从一段官方示例代码开始:
public class SyncProducer {
public static void main(String[] args) throws Exception {
// 初始化一个producer并设置Producer group name
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name"); //(1)
// 设置NameServer地址
producer.setNamesrvAddr("localhost:9876"); //(2)
// 启动producer
producer.start();
for (int i = 0; i < 100; i++) {
// 创建一条消息,并指定topic、tag、body等信息,tag可以理解成标签,对消息进行再归类,RocketMQ可以在消费端对tag进行过滤
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
); //(3)
// 利用producer进行发送,并同步等待发送结果
SendResult sendResult = producer.send(msg); //(4)
System.out.printf("%s%n", sendResult);
}
// 一旦producer不再使用,关闭producer
producer.shutdown();
}
}
同步发送的整个代码流程如下:
- 首先会创建一个producer。普通消息可以创建 DefaultMQProducer,创建时需要填写生产组的名称,生产者组是指同一类Producer的集合,这类Producer发送同一类消息且发送逻辑一致。
- 设置 NameServer 的地址。Apache RocketMQ很多方式设置NameServer地址(客户端配置中有介绍),这里是在代码中调用producer的API setNamesrvAddr进行设置,如果有多个NameServer,中间以分号隔开,比如"127.0.0.2:9876;127.0.0.3:9876"。
- 第三步是构建消息。指定topic、tag、body等信息,tag可以理解成标签,对消息进行再归类,RocketMQ可以在消费端对tag进行过滤。
- 最后调用send接口将消息发送出去。同步发送等待结果最后返回SendResult,SendResut包含实际发送状态还包括SEND_OK(发送成功), FLUSH_DISK_TIMEOUT(刷盘超时), FLUSH_SLAVE_TIMEOUT(同步到备超时), SLAVE_NOT_AVAILABLE(备不可用),如果发送失败会抛出异常。
消息生产的入口类是DefaultMQProducer,首先要调用start()启动生产者实例,然后调用send()发送消息,这就算结束了,消息发送完成。是不是非常简单,接下来,我们来窥探一下它的内部实现原理。
2.1.1 DefaultMQProducer
2.1.1.1 类的继承关系
继承ClientConfig,实现MQProducer接口(定义了,生产者所有的功能实现),这个类是对外给客户使用的,实际的内部实现都在DefaultMQProducerImpl这个类中,稍后会讲到。下面是继承关系图(偷个懒,网上找的)
2.1.1.2 构造函数
// 默认构造函数
public DefaultMQProducer() {
// 指定默认的生产者组为:DEFAULT_PRODUCER
this(null, MixAll.DEFAULT_PRODUCER_GROUP, null);
}
/**
* 指定 RPC hook 的构造函数
*
* @param rpcHook RPC hook 用于每次执行远程处理命令.
*/
public DefaultMQProducer(RPCHook rpcHook) {
this(null, MixAll.DEFAULT_PRODUCER_GROUP, rpcHook);
}
// 指定生产者组的构造函数
public DefaultMQProducer(final String producerGroup) {
this(null, producerGroup, null);
}
public DefaultMQProducer(final String producerGroup, RPCHook rpcHook, boolean enableMsgTrace,
final String customizedTraceTopic) {
this.producerGroup = producerGroup; // 指定生产者组
// 内部消息生产者实现类,详情看章节`2.1.2`
defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
//是否客户端开启消息追踪
if (enableMsgTrace) {
try {
// 下面逻辑都是涉及消息追踪的,这个留待后续章节讲
AsyncTraceDispatcher dispatcher = new AsyncTraceDispatcher(customizedTraceTopic, rpcHook);
dispatcher.setHostProducer(this.defaultMQProducerImpl);
traceDispatcher = dispatcher;
this.defaultMQProducerImpl.registerSendMessageHook(
new SendMessageTraceHookImpl(traceDispatcher));
} catch (Throwable e) {
log.error("system mqtrace hook init failed ,maybe can't send msg trace data");
}
}
}
// 指定命名空间和生者组的构造函数
public DefaultMQProducer(final String namespace, final String producerGroup) {
this(namespace, producerGroup, null);
}
// 指定 rpcHook 和生者组的构造函数
public DefaultMQProducer(final String producerGroup, RPCHook rpcHook) {
this(null, producerGroup, rpcHook);
}
// 指定 rpcHook、生者组和命名空间的构造函数
public DefaultMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook) {
this.namespace = namespace; // 命名空间
this.producerGroup = producerGroup; // 生产者组
// 内部消息生产者实现类,详情看章节`2.1.2`
defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
}
// 指定生产者组和消息追踪的构造函数
public DefaultMQProducer(final String producerGroup, boolean enableMsgTrace) {
this(null, producerGroup, null, enableMsgTrace, null);
}
// 指定生产者组、消息追踪和追踪topic的构造函数
public DefaultMQProducer(final String producerGroup, boolean enableMsgTrace, final String customizedTraceTopic) {
this(null, producerGroup, null, enableMsgTrace, customizedTraceTopic);
}
public DefaultMQProducer(final String namespace, final String producerGroup, RPCHook rpcHook,
boolean enableMsgTrace, final String customizedTraceTopic) {
this.namespace = namespace; // 命名空间
this.producerGroup = producerGroup; // 生产者组
// 内部消息生产者实现类,详情看章节`2.1.2`
defaultMQProducerImpl = new DefaultMQProducerImpl(this, rpcHook);
//是否客户端开启消息追踪
if (enableMsgTrace) {
try {
// 下面逻辑都是涉及消息追踪的,这个留待后续章节讲
AsyncTraceDispatcher dispatcher = new AsyncTraceDispatcher(customizedTraceTopic, rpcHook);
dispatcher.setHostProducer(this.getDefaultMQProducerImpl());
traceDispatcher = dispatcher;
this.getDefaultMQProducerImpl().registerSendMessageHook(
new SendMessageTraceHookImpl(traceDispatcher));
} catch (Throwable e) {
log.error("system mqtrace hook init failed ,maybe can't send msg trace data");
}
}
}
2.1.1.3 start启动
public void start() throws MQClientException {
// 将生产者组改成:namespace + '%' + producerGroup,当然里面还有一些针对重试和死信消息的处理,有兴趣的读者可以进去看看
this.setProducerGroup(withNamespace(this.producerGroup));
this.defaultMQProducerImpl.start(); //DefaultMQProducerImpl具体实现启动逻辑,详情看章节`2.1.2`
if (null != traceDispatcher) {
try {
// 启动消息追踪
traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
} catch (MQClientException e) {
log.warn("trace dispatcher start failed ", e);
}
}
}
2.1.1.4 send发送消息
public SendResult send( Message msg)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
/*消费的校验,主要校验以下信息:
1.topic判空、格式、长度(255)、是否为TBW102,只要1条不合格,校验失败
2.body消息体判空,如果为空,抛出异常
3.body消息体长度判断:默认4M,这个值可以修改,但是建议不要改,超过抛出异常
*/
Validators.checkMessage(msg, this);
//如果有命名空间的,那么要在原topic前增加命名空间的前缀:namespace + '%' + topic
msg.setTopic(withNamespace(msg.getTopic()));
//DefaultMQProducerImpl具体实现启动逻辑,详情看章节`2.1.2`
return this.defaultMQProducerImpl.send(msg);
}
2.1.2 DefaultMQProducerImpl
2.1.2.1 start实现逻辑
public void start(final boolean startFactory) throws MQClientException {
// 服务状态:CREATE_JUST--刚创建、RUNNING--运行中、SHUTDOWN_ALREADY--服务关闭、START_FAILED--启动失败
switch (this.serviceState) {
case CREATE_JUST: //创建还未启动
this.serviceState = ServiceState.START_FAILED;
//group命名检查,主要检查空、格式、长度,并且不能是默认生产者组名
this.checkConfig();
//判断group名字与CLIENT_INNER_PRODUCER不等,证明不是内部group,而是用户自定义的group
if (!this.defaultMQProducer.getProducerGroup().equals(MixAll.CLIENT_INNER_PRODUCER_GROUP)) {
this.defaultMQProducer.changeInstanceNameToPID();//将实例进程id设置成实例名
}
//获取或创建(详情见`3.2.1`)MQClientInstance实例,该实例通过单例模式获取,这个类很重要,同时涵盖了生产者和消费者的操作,详情见3.3
this.mQClientFactory = MQClientManager.getInstance()
.getAndCreateMQClientInstance(this.defaultMQProducer, rpcHook);
//将当前生产者实例对象与group名绑定,并注册到MQClientInstance对象,方便后续使用时通过group取出,见`3.3.1`
boolean registerOK = mQClientFactory
.registerProducer(this.defaultMQProducer.getProducerGroup(), this);
if (!registerOK) {
//重复注册失败,恢复状态,并抛出异常
this.serviceState = ServiceState.CREATE_JUST;
throw new MQClientException("The producer group[" + this.defaultMQProducer.getProducerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
//创建topic发布消息对象TopicPublishInfo,并存入topicPublishInfoTable map容器中
this.topicPublishInfoTable.put(this.defaultMQProducer.getCreateTopicKey(), new TopicPublishInfo());
if (startFactory) {
//重点在这里,前面都是准备工作,这里才是真正开始工作的地方,详情见`3.3.2`
mQClientFactory.start();
}
log.info("the producer [{}] start OK. sendMessageWithVIPChannel={}", this.defaultMQProducer.getProducerGroup(),
this.defaultMQProducer.isSendMessageWithVIPChannel());
// 启动成功,状态变更为RUNNING
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
// 这三种状态下重复启动,抛出异常
throw new MQClientException("The producer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
// 给所有Broker发送心跳,这块内容后续也会讲
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
}
/**
1.group判空
2.group命名规范判断
3.group长度判断(255)
4.group默认名字(DEFAULT_PRODUCER)相等判断
*/
private void checkConfig() throws MQClientException {
//校验group是否为空、命名规范、长度(255个字符),所以大家命名时要注意规范和长度
Validators.checkGroup(this.defaultMQProducer.getProducerGroup());
//再次判null,感觉这一步有点多余,估计是开发人员写顺手了
if (null == this.defaultMQProducer.getProducerGroup()) {
throw new MQClientException("producerGroup is null", null);
}
//group名字不要命名为默认名字“DEFAULT_PRODUCER”
if (this.defaultMQProducer.getProducerGroup().equals(MixAll.DEFAULT_PRODUCER_GROUP)) {
throw new MQClientException("producerGroup can not equal " + MixAll.DEFAULT_PRODUCER_GROUP + ", please specify another one.", null);
}
}
2.1.2.2 send发送消息
public SendResult send(
Message msg) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
// 默认发送消息超时3秒钟
return send(msg, this.defaultMQProducer.getSendMsgTimeout());
}
public SendResult send(Message msg,
long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
// 同步发送消息,枚举 CommunicationMode 定义三种类型:同步、异步、单向发送
return this.sendDefaultImpl(msg, CommunicationMode.SYNC, null, timeout);
}
sendDefaultImpl
/*
1.msg:要发送的消息
2.communicationMode:发送模式(SYNC-同步、ASYNC-异步、ONEWAY-单次)
3.sendCallback:发送后回调
4.timeout:发送等待超时
*/
private SendResult sendDefaultImpl(
Message msg,
final CommunicationMode communicationMode,
final SendCallback sendCallback,
final long timeout
) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
//校验状态,如果不是RUNNING状态,直接抛出异常
this.makeSureStateOK();
/*消费的校验,主要校验以下信息:
1.topic判空、格式、长度(255)、是否为TBW102,只要1条不合格,校验失败
2.body消息体判空,如果为空,抛出异常
3.body消息体长度判断:默认4M,这个值可以修改,但是建议不要改,超过抛出异常
*/
Validators.checkMessage(msg, this.defaultMQProducer);
//随机生成一个long值作为调用id
final long invokeID = random.nextLong();
//记录开始和结束时间
long beginTimestampFirst = System.currentTimeMillis();
long beginTimestampPrev = beginTimestampFirst;
long endTimestamp = beginTimestampFirst;
//从内存map topicPublishInfoTable中尝试获取发布信息,如果没有,则创建一个新的,并从name server拉取路由信息,填充内容到新创建的 TopicPublishInfo
TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic());
//拿到发布消息内容,并且状态ok(ok的依据是messageQueueList不为空),来源于哪里呢,其实他来自于创建topic时设置的读写队列数,可以从章节`3.3.4`中窥探。messageQueueList 肯定是不能为空的,要是为空,后面的流程就走不下去了。
if (topicPublishInfo != null && topicPublishInfo.ok()) {
boolean callTimeout = false;
MessageQueue mq = null;
Exception exception = null;
SendResult sendResult = null;
//失败尝试次数,只针对同步发送,默认值是2,再加上1,就是3了,其他模式(异步、单次)下只发送1次
int timesTotal = communicationMode == CommunicationMode.SYNC ? 1 + this.defaultMQProducer.getRetryTimesWhenSendFailed() : 1;
int times = 0;
//记录每次使用的brokerName
String[] brokersSent = new String[timesTotal];
//遍历次数
for (; times < timesTotal; times++) {
//记录上一次发送的brokerName,初次发送时为null
String lastBrokerName = null == mq ? null : mq.getBrokerName();
//选择一个消息队列来发送,选择的策略是遍历取模,发送失败后,下一次发送,会选择与上一次一样的mq
MessageQueue mqSelected = this.selectOneMessageQueue(topicPublishInfo, lastBrokerName);
if (mqSelected != null) {
mq = mqSelected;
brokersSent[times] = mq.getBrokerName();
try {
beginTimestampPrev = System.currentTimeMillis();
if (times > 0) {
// times 大于 0,表示进入重试了,此时的topic要增加namespace前缀了
msg.setTopic(this.defaultMQProducer.withNamespace(msg.getTopic()));
}
long costTime = beginTimestampPrev - beginTimestampFirst;
//超时了,直接退出遍历
if (timeout < costTime) {
callTimeout = true;
break;
}
//消息发送,并返回结果
sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, topicPublishInfo, timeout - costTime);
endTimestamp = System.currentTimeMillis();
// 记录当前使用的broker信息及时间
this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
switch (communicationMode) {
case ASYNC:
return null; // 异步发送不适用这种方式,直接返回 Null
case ONEWAY:
return null; // 单向发送也不适用这种方式,直接返回 Null
case SYNC:
if (sendResult.getSendStatus() != SendStatus.SEND_OK) {
//发送失败时,会先判断是否设置了当失败时重试其他broker的标志,如是,继续下一次重试,否则直接返回发送结果
if (this.defaultMQProducer.isRetryAnotherBrokerWhenNotStoreOK()) {
continue;
}
}
// 发送成功,返回发送返回结果
return sendResult;
default:
break;
}
} catch (RemotingException e) { // 下面都是针对各种异常情况做的日志记录和打印,且都会调用 updateFaultItem 方法,在缓存map中记录失败的broker和时间,供下次选择合适的broker做参考
endTimestamp = System.currentTimeMillis();
this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
log.warn(msg.toString());
exception = e;
continue;
} catch (MQClientException e) {
endTimestamp = System.currentTimeMillis();
this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
log.warn(msg.toString());
exception = e;
continue;
} catch (MQBrokerException e) {
endTimestamp = System.currentTimeMillis();
this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, true);
log.warn(String.format("sendKernelImpl exception, resend at once, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
log.warn(msg.toString());
exception = e;
switch (e.getResponseCode()) {
case ResponseCode.TOPIC_NOT_EXIST:
case ResponseCode.SERVICE_NOT_AVAILABLE:
case ResponseCode.SYSTEM_ERROR:
case ResponseCode.NO_PERMISSION:
case ResponseCode.NO_BUYER_ID:
case ResponseCode.NOT_IN_CURRENT_UNIT:
continue;
default:
if (sendResult != null) {
return sendResult;
}
throw e;
}
} catch (InterruptedException e) {
endTimestamp = System.currentTimeMillis();
this.updateFaultItem(mq.getBrokerName(), endTimestamp - beginTimestampPrev, false);
log.warn(String.format("sendKernelImpl exception, throw exception, InvokeID: %s, RT: %sms, Broker: %s", invokeID, endTimestamp - beginTimestampPrev, mq), e);
log.warn(msg.toString());
log.warn("sendKernelImpl exception", e);
log.warn(msg.toString());
throw e;
}
} else {
//没有选择到mq,直接退出循环
break;
}
}
// 这里其实就是多重判断
if (sendResult != null) {
return sendResult;
}
//sendResult为空,就是没有选择到可发送的mq,接下来告知错误,并记录相关日志
String info = String.format("Send [%d] times, still failed, cost [%d]ms, Topic: %s, BrokersSent: %s",
times,
System.currentTimeMillis() - beginTimestampFirst,
msg.getTopic(),
Arrays.toString(brokersSent));
info += FAQUrl.suggestTodo(FAQUrl.SEND_MSG_FAILED);
MQClientException mqClientException = new MQClientException(info, exception);
if (callTimeout) {
throw new RemotingTooMuchRequestException("sendDefaultImpl call timeout");
}
if (exception instanceof MQBrokerException) {
mqClientException.setResponseCode(((MQBrokerException) exception).getResponseCode());
} else if (exception instanceof RemotingConnectException) {
mqClientException.setResponseCode(ClientErrorCode.CONNECT_BROKER_EXCEPTION);
} else if (exception instanceof RemotingTimeoutException) {
mqClientException.setResponseCode(ClientErrorCode.ACCESS_BROKER_TIMEOUT);
} else if (exception instanceof MQClientException) {
mqClientException.setResponseCode(ClientErrorCode.BROKER_NOT_EXIST_EXCEPTION);
}
throw mqClientException;
}
List<String> nsList = this.getmQClientFactory().getMQClientAPIImpl().getNameServerAddressList();
if (null == nsList || nsList.isEmpty()) {
throw new MQClientException(
"No name server address, please set it." + FAQUrl.suggestTodo(FAQUrl.NAME_SERVER_ADDR_NOT_EXIST_URL), null).setResponseCode(ClientErrorCode.NO_NAME_SERVER_EXCEPTION);
}
throw new MQClientException("No route info of this topic, " + msg.getTopic() + FAQUrl.suggestTodo(FAQUrl.NO_TOPIC_ROUTE_INFO),
null).setResponseCode(ClientErrorCode.NOT_FOUND_TOPIC_EXCEPTION);
}
sendKernelImpl
具体做消息发送的方法
private SendResult sendKernelImpl(final Message msg,
final MessageQueue mq,
final CommunicationMode communicationMode,
final SendCallback sendCallback,
final TopicPublishInfo topicPublishInfo,
final long timeout) throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
long beginStartTime = System.currentTimeMillis();
// 根据 brokerName 从本地缓存中获取 broker 地址
String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
if (null == brokerAddr) {
// 本地缓存中没有,尝试从 name server 远程拉取
tryToFindTopicPublishInfo(mq.getTopic());
brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName());
}
SendMessageContext context = null;
if (brokerAddr != null) {
// 这里会根据是否用vip通道来修改 broker 的地址,实际就是ip不换,端口改成原来端口减2
brokerAddr = MixAll.brokerVIPChannel(this.defaultMQProducer.isSendMessageWithVIPChannel(), brokerAddr);
// 发送的消息体
byte[] prevBody = msg.getBody();
try {
//for MessageBatch,ID has been set in the generating process
//判断是不是批量发送消息,不是的话要设置一个唯一id
if (!(msg instanceof MessageBatch)) {
MessageClientIDSetter.setUniqID(msg);
}
boolean topicWithNamespace = false;
if (null != this.mQClientFactory.getClientConfig().getNamespace()) {
// 将 instanceId 的值设置为 namespace
msg.setInstanceId(this.mQClientFactory.getClientConfig().getNamespace());
topicWithNamespace = true; // 标志是带命名空间的topic
}
int sysFlag = 0;
boolean msgBodyCompressed = false;
//针对超过消息长度(4K)限制的消息,尝试压缩
if (this.tryToCompressMessage(msg)) {
// 标志消息是经过压缩的
sysFlag |= MessageSysFlag.COMPRESSED_FLAG;
msgBodyCompressed = true;
}
//判断系统标志,这里主要是判断是不是事务消息,并设置成事务预处理类型,因为事务消息的保障类似于Mysql的事务,是靠两段提交模式实现
final String tranMsg = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (tranMsg != null && Boolean.parseBoolean(tranMsg)) {
sysFlag |= MessageSysFlag.TRANSACTION_PREPARED_TYPE;
}
//接下来都是一些钩子实现,可由用户自定义钩子类,由于该版本的rocketmq,没有默认实现,这块知识点,咱们就不讲了,有兴趣的读者可以自行实现并实验
if (hasCheckForbiddenHook()) {
CheckForbiddenContext checkForbiddenContext = new CheckForbiddenContext();
checkForbiddenContext.setNameSrvAddr(this.defaultMQProducer.getNamesrvAddr());
checkForbiddenContext.setGroup(this.defaultMQProducer.getProducerGroup());
checkForbiddenContext.setCommunicationMode(communicationMode);
checkForbiddenContext.setBrokerAddr(brokerAddr);
checkForbiddenContext.setMessage(msg);
checkForbiddenContext.setMq(mq);
checkForbiddenContext.setUnitMode(this.isUnitMode());
this.executeCheckForbiddenHook(checkForbiddenContext);
}
// 消息发送追踪信息trace的hook,这块 rokcetmq 是有默认实现的,后续章节会讲
if (this.hasSendMessageHook()) {
context = new SendMessageContext();
context.setProducer(this);
context.setProducerGroup(this.defaultMQProducer.getProducerGroup());
context.setCommunicationMode(communicationMode);
context.setBornHost(this.defaultMQProducer.getClientIP());
context.setBrokerAddr(brokerAddr);
context.setMessage(msg);
context.setMq(mq);
context.setNamespace(this.defaultMQProducer.getNamespace());
String isTrans = msg.getProperty(MessageConst.PROPERTY_TRANSACTION_PREPARED);
if (isTrans != null && isTrans.equals("true")) {
context.setMsgType(MessageType.Trans_Msg_Half);
}
if (msg.getProperty("__STARTDELIVERTIME") != null || msg.getProperty(MessageConst.PROPERTY_DELAY_TIME_LEVEL) != null) {
context.setMsgType(MessageType.Delay_Msg);
}
this.executeSendMessageHookBefore(context);
}
//封装发送请求消息头
SendMessageRequestHeader requestHeader = new SendMessageRequestHeader();
requestHeader.setProducerGroup(this.defaultMQProducer.getProducerGroup());
requestHeader.setTopic(msg.getTopic());
requestHeader.setDefaultTopic(this.defaultMQProducer.getCreateTopicKey());
requestHeader.setDefaultTopicQueueNums(this.defaultMQProducer.getDefaultTopicQueueNums());
requestHeader.setQueueId(mq.getQueueId());
requestHeader.setSysFlag(sysFlag);
requestHeader.setBornTimestamp(System.currentTimeMillis());
requestHeader.setFlag(msg.getFlag());
requestHeader.setProperties(MessageDecoder.messageProperties2String(msg.getProperties()));
requestHeader.setReconsumeTimes(0);
requestHeader.setUnitMode(this.isUnitMode());
requestHeader.setBatch(msg instanceof MessageBatch);
//这里为什么会判断重试消息呢?其实这里主要用在消费者端,当消息失败的时候是消费者客户端自己主动重新推送消息到broker,但是此时的topic就要以"%RETRY%"开头,并拼接原topic信息
if (requestHeader.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
String reconsumeTimes = MessageAccessor.getReconsumeTime(msg);
if (reconsumeTimes != null) {
requestHeader.setReconsumeTimes(Integer.valueOf(reconsumeTimes));
MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_RECONSUME_TIME);
}
String maxReconsumeTimes = MessageAccessor.getMaxReconsumeTimes(msg);
if (maxReconsumeTimes != null) {
requestHeader.setMaxReconsumeTimes(Integer.valueOf(maxReconsumeTimes));
MessageAccessor.clearProperty(msg, MessageConst.PROPERTY_MAX_RECONSUME_TIMES);
}
}
SendResult sendResult = null;
switch (communicationMode) {
case ASYNC:
Message tmpMessage = msg;
boolean messageCloned = false;
if (msgBodyCompressed) {
//If msg body was compressed, msgbody should be reset using prevBody.
//Clone new message using commpressed message body and recover origin massage.
//Fix bug:https://github.com/apache/rocketmq-externals/issues/66
tmpMessage = MessageAccessor.cloneMessage(msg);
messageCloned = true;
msg.setBody(prevBody);
}
if (topicWithNamespace) {
if (!messageCloned) {
tmpMessage = MessageAccessor.cloneMessage(msg);
messageCloned = true;
}
msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQProducer.getNamespace()));
}
long costTimeAsync = System.currentTimeMillis() - beginStartTime;
if (timeout < costTimeAsync) {
throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
}
// 异步消息,这里需要回调方法
sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
brokerAddr,
mq.getBrokerName(),
tmpMessage,
requestHeader,
timeout - costTimeAsync,
communicationMode,
sendCallback,
topicPublishInfo,
this.mQClientFactory,
this.defaultMQProducer.getRetryTimesWhenSendAsyncFailed(),
context,
this);
break;
case ONEWAY:
case SYNC:
long costTimeSync = System.currentTimeMillis() - beginStartTime;
if (timeout < costTimeSync) {
throw new RemotingTooMuchRequestException("sendKernelImpl call timeout");
}
//消息发送,内部是通过Netty来传输的,看章节`3.5.1`
sendResult = this.mQClientFactory.getMQClientAPIImpl().sendMessage(
brokerAddr,
mq.getBrokerName(),
msg,
requestHeader,
timeout - costTimeSync,
communicationMode,
context,
this);
break;
default:
assert false;
break;
}
if (this.hasSendMessageHook()) {
context.setSendResult(sendResult);
this.executeSendMessageHookAfter(context);
}
return sendResult;
} catch (RemotingException e) {
if (this.hasSendMessageHook()) {
context.setException(e);
this.executeSendMessageHookAfter(context);
}
throw e;
} catch (MQBrokerException e) {
if (this.hasSendMessageHook()) {
context.setException(e);
this.executeSendMessageHookAfter(context);
}
throw e;
} catch (InterruptedException e) {
if (this.hasSendMessageHook()) {
context.setException(e);
this.executeSendMessageHookAfter(context);
}
throw e;
} finally {
msg.setBody(prevBody);
msg.setTopic(NamespaceUtil.withoutNamespace(msg.getTopic(), this.defaultMQProducer.getNamespace()));
}
}
throw new MQClientException("The broker[" + mq.getBrokerName() + "] not exist", null);
}
2.1.3 消息发送队列选择策略
MQFaultStrategy.selectOneMessageQueue
public MessageQueue selectOneMessageQueue(final TopicPublishInfo tpInfo, final String lastBrokerName) {
if (this.sendLatencyFaultEnable) { // 发送延迟故障标致是否打开,主要用在延迟发送和消息发送失败重试
try {
// 每次发送时都会+1
int index = tpInfo.getSendWhichQueue().getAndIncrement();
// 遍历消息队列
for (int i = 0; i < tpInfo.getMessageQueueList().size(); i++) {
// 这里用 index 取模,Math.abs(index++) 为啥要用绝对值呢?因为在计算机中整型一直增加就可能变成负数,所以要用绝对值
int pos = Math.abs(index++) % tpInfo.getMessageQueueList().size();
if (pos < 0)
pos = 0;
// 取出对应索引的 MessageQueue
MessageQueue mq = tpInfo.getMessageQueueList().get(pos);
// 判断当前mq是否可用,可用就直接返回mq
if (latencyFaultTolerance.isAvailable(mq.getBrokerName())) {
if (null == lastBrokerName || mq.getBrokerName().equals(lastBrokerName))
return mq;
}
}
// 上面方式找不到合适的,只能重新洗牌,找一个不是最好的broker
final String notBestBroker = latencyFaultTolerance.pickOneAtLeast();
// 获取该 broker 的写队列数
int writeQueueNums = tpInfo.getQueueIdByBroker(notBestBroker);
if (writeQueueNums > 0) {// 队列数大于0
// 依然按取模求出一个队列
final MessageQueue mq = tpInfo.selectOneMessageQueue();
// notBestBroker 不为null 判断
if (notBestBroker != null) {
// 将当前 mq 的broker 换成 notBestBroker
mq.setBrokerName(notBestBroker);
// 队列id,依然通过取模来获取
mq.setQueueId(tpInfo.getSendWhichQueue().getAndIncrement() % writeQueueNums);
}
return mq;
} else {
// 没有写队列,证明该broker无法使用,则直接从缓存map中移除该 broker
latencyFaultTolerance.remove(notBestBroker);
}
} catch (Exception e) {
log.error("Error occurred when selecting message queue", e);
}
return tpInfo.selectOneMessageQueue();
}
// 一般情况下,都是走到这里,调用 TopicPublishInfo.selectOneMessageQueue
return tpInfo.selectOneMessageQueue(lastBrokerName);
}
TopicPublishInfo.selectOneMessageQueue
public MessageQueue selectOneMessageQueue(final String lastBrokerName) {
if (lastBrokerName == null) { // 初次调用,走这里
return selectOneMessageQueue();
} else { // 第一次失败,重试时,走这里
int index = this.sendWhichQueue.getAndIncrement();
for (int i = 0; i < this.messageQueueList.size(); i++) {
int pos = Math.abs(index++) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
MessageQueue mq = this.messageQueueList.get(pos);
// 也是取模,只是增加了对 lastBrokerName 的判断,也就是说失败了,重试时就不会再用这个失败的 broker下的队列,得重新找一个不同名的
if (!mq.getBrokerName().equals(lastBrokerName)) {
return mq;
}
}
// 这里做一个兜底,试想一下,假如就一个broker呢,那只能从这里取 mq 了
return selectOneMessageQueue();
}
}
public MessageQueue selectOneMessageQueue() {
// 增加 index 值,并取模,求出消息队列中对应的 mq
int index = this.sendWhichQueue.getAndIncrement();
int pos = Math.abs(index) % this.messageQueueList.size();
if (pos < 0)
pos = 0;
return this.messageQueueList.get(pos);
}
2.2 异步发送
异步发送是指发送方发出一条消息后,不等服务端返回响应,接着发送下一条消息的通讯方式。
先看个例子:
public class AsyncProducer {
public static void main(String[] args) throws Exception {
// 初始化一个producer并设置Producer group name
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer地址
producer.setNamesrvAddr("localhost:9876");
// 启动producer
producer.start();
producer.setRetryTimesWhenSendAsyncFailed(0);
int messageCount = 100;
final CountDownLatch countDownLatch = new CountDownLatch(messageCount);
for (int i = 0; i < messageCount; i++) {
try {
final int index = i;
// 创建一条消息,并指定topic、tag、body等信息,tag可以理解成标签,对消息进行再归类,RocketMQ可以在消费端对tag进行过滤
Message msg = new Message("TopicTest",
"TagA",
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
// 异步发送消息, 发送结果通过callback返回给客户端
producer.send(msg, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
System.out.printf("%-10d OK %s %n", index,
sendResult.getMsgId());
countDownLatch.countDown();
}
@Override
public void onException(Throwable e) {
System.out.printf("%-10d Exception %s %n", index, e);
e.printStackTrace();
countDownLatch.countDown();
}
});
} catch (Exception e) {
e.printStackTrace();
countDownLatch.countDown();
}
}
//异步发送,如果要求可靠传输,必须要等回调接口返回明确结果后才能结束逻辑,否则立即关闭Producer可能导致部分消息尚未传输成功
countDownLatch.await(5, TimeUnit.SECONDS);
// 一旦producer不再使用,关闭producer
producer.shutdown();
}
}
2.2.1 DefaultMQProducer
2.2.1.1 send发送消息
public void send(Message msg,
SendCallback sendCallback) throws MQClientException, RemotingException, InterruptedException {
// 用命名空间重新包装topic,并设置成新topic去执行后续流程
msg.setTopic(withNamespace(msg.getTopic()));
//DefaultMQProducerImpl具体实现启动逻辑,详情看章节`2.2.2.1`
this.defaultMQProducerImpl.send(msg, sendCallback);
}
@Override
public void send(Message msg, SendCallback sendCallback, long timeout)
throws MQClientException, RemotingException, InterruptedException {
// 用命名空间重新包装topic,并设置成新topic去执行后续流程
msg.setTopic(withNamespace(msg.getTopic()));
//DefaultMQProducerImpl具体实现启动逻辑,详情看章节`2.2.2.1`
this.defaultMQProducerImpl.send(msg, sendCallback, timeout);
}
2.2.2 DefaultMQProducerImpl
2.2.2.1 send发送消息
public void send(Message msg,
SendCallback sendCallback) throws MQClientException, RemotingException, InterruptedException {
send(msg, sendCallback, this.defaultMQProducer.getSendMsgTimeout());
}
@Deprecated
public void send(final Message msg, final SendCallback sendCallback, final long timeout)
throws MQClientException, RemotingException, InterruptedException {
final long beginStartTime = System.currentTimeMillis();
// 这里用线程池来实现异步任务的执行,默认情况下,RocketMQ 用的是ThreadPoolExecutor来实现的,关于线程池,有兴趣的读者,可以看我的另一篇博文:https://blog.csdn.net/zhang527294844/article/details/134139332
ExecutorService executor = this.getAsyncSenderExecutor();
try {
// 将任务提交到线程池
executor.submit(new Runnable() {
@Override
public void run() {
long costTime = System.currentTimeMillis() - beginStartTime;
if (timeout > costTime) {
try {
// 最终还是会调用底层的发送方法,这个方法在章节`2.1.2.2`已经讲过,可以回头去看看,另外,对异步回调的方法,可以看章节`3.5.2`
sendDefaultImpl(msg, CommunicationMode.ASYNC, sendCallback, timeout - costTime);
} catch (Exception e) {
sendCallback.onException(e);
}
} else {
sendCallback.onException(
new RemotingTooMuchRequestException("DEFAULT ASYNC send call timeout"));
}
}
});
} catch (RejectedExecutionException e) {
throw new MQClientException("executor rejected ", e);
}
}
2.3 单向发送
发送方只负责发送消息,不等待服务端返回响应且没有回调函数触发,即只发送请求不等待应答。此方式发送消息的过程耗时非常短,一般在微秒级别。适用于某些耗时非常短,但对可靠性要求并不高的场景,例如日志收集。
先从一个示例开始:
public class OnewayProducer {
public static void main(String[] args) throws Exception{
// 初始化一个producer并设置Producer group name
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
// 设置NameServer地址
producer.setNamesrvAddr("localhost:9876");
// 启动producer
producer.start();
for (int i = 0; i < 100; i++) {
// 创建一条消息,并指定topic、tag、body等信息,tag可以理解成标签,对消息进行再归类,RocketMQ可以在消费端对tag进行过滤
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
// 由于在oneway方式发送消息时没有请求应答处理,如果出现消息发送失败,则会因为没有重试而导致数据丢失。若数据不可丢,建议选用可靠同步或可靠异步发送方式。
producer.sendOneway(msg);
}
// 一旦producer不再使用,关闭producer
producer.shutdown();
}
}
单向模式调用sendOneway,不会对返回结果有任何等待和处理,内部最终也是会调用到DefaultMQProducerImpl.sendDefaultImpl方法的,具体实现,看章节2.1.2.2