概述
前面讲了消息在broker端的存储,下面讲消息在消费者端的消费。简单介绍几个消费相关的概念。
集群消费&广播消费
是消费者两种不同的消费模式,广播消费模式下,一个消息会被所有的机器消费。集群消费模式下,一个消息只会被同一个集群消费一次,这里的集群由ConsumerGroup相同的机器组成。
ConsumerGroup
消费者归属于一个ConsumerGroup,相同ConsumerGroup的消费者组成一个集群,一个消息只被同一个集群的机器消费一次,起到负载均衡的作用。队列的消费进度也按照ConsumerGroup进行管理,不同ConsumerGroup之间的消费进度不受影响。
消费offset
用来保存消费进度,集群模式消费进度保存在broker,广播模式保存在消费者本地。
消息消费方式
分为push和pull两种方式。pull为消费者向broker主动拉取消息,可以根据负载控制消费速度。push为broker将消息推送到消费者,可以做到消息实时推送。在RocketMQ中,push模式是通过pull模式实现的。这里以push模式的实现作为分析。
相关类
- DefaultMQPushConsumer 消费客户端
- DefaultMQPushConsumerImpl 默认push消费者实现
- RebalanceImpl 负载均衡
- MQClientInstance 客户端实例
- PullAPIWrapper 拉取消息api
- OffsetStore 消费offset存储
消费者启动
DefaultMQPushConsumer.start()为启动入口,调用DefaultMQPushConsumerImpl.start():
public synchronized void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST: // 默认值为CREATE_JUST
log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
this.serviceState = ServiceState.START_FAILED;
// 检查相关配置
this.checkConfig();
// 复制订阅信息给RebalanceImpl负载均衡使用
this.copySubscription();
if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPushConsumer.changeInstanceNameToPID();
}
// 创建 MQClientInstance
this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
// 负载均衡设置
this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
// OffsetStore c初始化,分为集群和广播模式
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
case CLUSTERING:
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
default:
break;
}
this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
this.offsetStore.load();
// consumeMessageService 消费消息服务初始化
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
this.consumeOrderly = true;
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
this.consumeOrderly = false;
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
this.consumeMessageService.start();
// 注册消费者
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown();
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
// 更新订阅信息
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
this.mQClientFactory.checkClientInBroker();
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
// 立即执行负载均衡
this.mQClientFactory.rebalanceImmediately();
}
到这里启动完成。
拉取消息
拉取消息的入口在哪里呢?上面启动的时候,在MQClientInstanced.start()中启动了PullMessageService。PullMessageService继承自熟悉的ServiceThread,来看run方法:
public void run() {
while (!this.isStopped()) {
try {
// 从LinkedBlockingQueue取出请求,没有请求则阻塞
PullRequest pullRequest = this.pullRequestQueue.take();
// 执行拉取操作
this.pullMessage(pullRequest);
} catch (InterruptedException ignored) {
} catch (Exception e) {
log.error("Pull Message Service Run Method exception", e);
}
}
log.info(this.getServiceName() + " service end");
}
PullRequest放入:
- executePullRequestImmediately 立即放入,在消费者启动和拉取完成后放入
- executePullRequestLater 拉取异常情况下延迟放入
PullRequest类介绍
public class PullRequest {
// 消费组
private String consumerGroup;
// 消费队列信息
private MessageQueue messageQueue;
// 拉取到的消息存储
private ProcessQueue processQueue;
// 下次拉取offset
private long nextOffset;
}
PullMessageService拉取消息,交由DefaultMQPushConsumerImpl.pullMessage
private void pullMessage(final PullRequest pullRequest) {
final MQConsumerInner consumer = this.mQClientFactory.selectConsumer(pullRequest.getConsumerGroup());
if (consumer != null) {
// 转成 DefaultMQPushConsumerImpl,这个方法只为DefaultMQPushConsumerImpl服务,pull模式后面分析
DefaultMQPushConsumerImpl impl = (DefaultMQPushConsumerImpl) consumer;
impl.pullMessage(pullRequest);
} else {
log.warn("No matched consumer for the PullRequest {}, drop it", pullRequest);
}
}
DefaultMQPushConsumerImpl.pullMessage 拉取逻辑,交由pullAPIWrapper.pullKernelImpl
public void pullMessage(final PullRequest pullRequest) {
// 获取 消息存储队列
final ProcessQueue processQueue = pullRequest.getProcessQueue();
if (processQueue.isDropped()) {
log.info("the pull request[{}] is dropped.", pullRequest.toString());
return;
}
// 更新拉取时间戳
pullRequest.getProcessQueue().setLastPullTimestamp(System.currentTimeMillis