今天的博客主题
MQ消息中间件 --》RocketMQ --》RocketMQ进阶(二)
本文主要讲解RocketMQ消费者实现及底层实现原理刨根问底
同样和生产者一样,对于使用普通Java项目来实现的例子就不多说了,官方文档都有
样例地址:https://github.com/apache/rocketmq/blob/master/docs/cn/RocketMQ_Example.md
主要看下springboot实现消费者。
springboot集成rocketmq消费者实现
pom.xml文件引入MQ依赖jar包
<!-- rocketmq依赖 -->
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-spring-boot-starter</artifactId>
<version>2.0.3</version>
</dependency>
application.yml配置文件添加配置
rocketmq:
#服务地址
name-server: 10.10.16.20:9876
#消费者配置
consumer:
#消费组名
group: CG_DEMO
#主题
topic: TOPIC_DEMO_ONE
#标签
tag: TAG_ONE
具体代码实现
@Service
@RocketMQMessageListener(consumerGroup = "${rocketmq.consumer.group}", topic = "${rocketmq.consumer.topic}")
public class MsgListener implements RocketMQListener<MessageExt> {
private org.slf4j.Logger logger = LoggerFactory.getLogger(getClass());
@Override
public void onMessage(MessageExt msg) {
logger.debug("RECEIVE_MSG_BEGIN: " + msg.toString());
logger.debug(String.format("消费消息,消息ID:%s,消息KEY:%s,消息体:%s ", msg.getMsgId(), msg.getKeys(), new String(msg.getBody())));
}
}
测试
直接启动服务即可,会自动监听MQ服务,监听到消息就消费。
启动完就看到控制台输出:
2019-07-10 10:22:16.633 INFO 9504 --- [MessageThread_1] com.mq.listener.MsgListener : RECEIVE_MSG_BEGIN: MessageExt [queueId=2, storeSize=282, queueOffset=0, sysFlag=0, bornTimestamp=1562577172354, bornHost=/10.10.51.219:57247, storeTimestamp=1562576637320, storeHost=/10.10.16.20:10911, msgId=0A0A101400002A9F000000000006982F, commitLogOffset=432175, bodyCRC=1703141525, reconsumeTimes=0, preparedTransactionOffset=0, toString()=Message{topic='TOPIC_DEMO_ONE', flag=0, properties={MIN_OFFSET=0, MAX_OFFSET=1, KEYS=123456, CONSUME_START_TIME=1562725336633, id=6b3efbac-03e4-0ced-4a9d-a722be0f775c, UNIQ_KEY=0A0A33DB31DC18B4AAC227BE22DD0000, WAIT=false, TAGS=TAG_ONE, timestamp=1562577172027}, body=[72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100, 32, 50, 48, 49, 57, 45, 48, 55, 45, 48, 56, 84, 49, 55, 58, 49, 50, 58, 53, 50, 46, 48, 50, 50], transactionId='null'}]
2019-07-10 10:22:16.633 INFO 9504 --- [MessageThread_1] com.mq.listener.MsgListener : 消费消息,消息ID:0A0A33DB31DC18B4AAC227BE22DD0000,消息KEY:123456,消息体:Hello World 2019-07-08T17:12:52.022
到这里就在springboot中集成了rocketmq消费者并成功消费了生产者发送的消息。
下面开始对消费者消费消息进行刨根问底。
@RocketMQMessageListener 注解
在这个注解类里面,只有一些字符串类型的方法,主要来管控一些参数。
看一下这个注解是怎么初始化它的。看下被引用的地方,因为是自动装配的,我们直接找 Configuration 的哪个
ListenerContainerConfiguration 类,会在启动时候被spring扫描到
@Configuration
public class ListenerContainerConfiguration implements ApplicationContextAware, SmartInitializingSingleton {
// 实现了ApplicationContextAware 应用程序上下文感知 SmartInitializingSingleton 初始化单例
...省略部分代码
// 因为是实现必须要重写接口方法,直接从这个两个方法作为入口
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
this.applicationContext = (ConfigurableApplicationContext) applicationContext;
}
@Override
public void afterSingletonsInstantiated() {
// 获取带有 RocketMQMessageListener 的bean
Map<String, Object> beans = this.applicationContext.getBeansWithAnnotation(RocketMQMessageListener.class);
if (Objects.nonNull(beans)) {
beans.forEach(this::registerContainer);
}
}
// 初始化单利,将监听器注册到容器
private void registerContainer(String beanName, Object bean) {
Class<?> clazz = AopProxyUtils.ultimateTargetClass(bean);
// 是否可分配的实例
if (!RocketMQListener.class.isAssignableFrom(bean.getClass())) {
throw new IllegalStateException(clazz + " is not instance of " + RocketMQListener.class.getName());
}
// 获得消息监听器,进行验证
RocketMQMessageListener annotation = clazz.getAnnotation(RocketMQMessageListener.class);
validate(annotation);
// 容器bean名称
String containerBeanName = String.format("%s_%s", DefaultRocketMQListenerContainer.class.getName(),
counter.incrementAndGet());
GenericApplicationContext genericApplicationContext = (GenericApplicationContext) applicationContext;
// 将bean注册到通用上下文应用中
genericApplicationContext.registerBean(containerBeanName, DefaultRocketMQListenerContainer.class,
() -> createRocketMQListenerContainer(containerBeanName, bean, annotation));
// 从这个上下文中在获的这个bean?
DefaultRocketMQListenerContainer container = genericApplicationContext.getBean(containerBeanName,
DefaultRocketMQListenerContainer.class);
if (!container.isRunning()) {
try {
// 调用这个容器里的开始方法,就是启动consumer
container.start();
} catch (Exception e) {
log.error("Started container failed. {}", container, e);
throw new RuntimeException(e);
}
}
log.info("Register the listener to container, listenerBeanName:{}, containerBeanName:{}", beanName, containerBeanName);
}
}
通过 DefaultRocketMQListenerContainer 里的 initRocketMQPushConsumer 来初始化 DefaultMQPushConsumer
public class DefaultRocketMQListenerContainer implements InitializingBean, RocketMQListenerContainer, SmartLifecycle, ApplicationContextAware {
省略部分代码...
private void initRocketMQPushConsumer() throws MQClientException {
Assert.notNull(rocketMQListener, "Property 'rocketMQListener' is required");
Assert.notNull(consumerGroup, "Property 'consumerGroup' is required");
Assert.notNull(nameServer, "Property 'nameServer' is required");
Assert.notNull(topic, "Property 'topic' is required");
// 权限钩子
RPCHook rpcHook = RocketMQUtil.getRPCHookByAkSk(applicationContext.getEnvironment(),
this.rocketMQMessageListener.accessKey(), this.rocketMQMessageListener.secretKey());
boolean enableMsgTrace = rocketMQMessageListener.enableMsgTrace();
// 实例化DefaultMQPushConsumer
if (Objects.nonNull(rpcHook)) {
consumer = new DefaultMQPushConsumer(consumerGroup, rpcHook, new AllocateMessageQueueAveragely(),
enableMsgTrace, this.applicationContext.getEnvironment().
resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
consumer.setVipChannelEnabled(false);
consumer.setInstanceName(RocketMQUtil.getInstanceName(rpcHook, consumerGroup));
} else {
log.debug("Access-key or secret-key not configure in " + this + ".");
consumer = new DefaultMQPushConsumer(consumerGroup, enableMsgTrace,
this.applicationContext.getEnvironment().
resolveRequiredPlaceholders(this.rocketMQMessageListener.customizedTraceTopic()));
}
// 为DefaultMQPushConsumer设置上下文参数
String customizedNameServer = this.applicationContext.getEnvironment().resolveRequiredPlaceholders(this.rocketMQMessageListener.nameServer());
if (customizedNameServer != null) {
consumer.setNamesrvAddr(customizedNameServer);
} else {
consumer.setNamesrvAddr(nameServer);
}
if (accessChannel != null) {
consumer.setAccessChannel(accessChannel);
}
consumer.setConsumeThreadMax(consumeThreadMax);
if (consumeThreadMax < consumer.getConsumeThreadMin()) {
consumer.setConsumeThreadMin(consumeThreadMax);
}
consumer.setConsumeTimeout(consumeTimeout);
consumer.setInstanceName(this.name);
// 消费方式
switch (messageModel) {
case BROADCASTING:
consumer.setMessageModel(org.apache.rocketmq.common.protocol.heartbeat.MessageModel.BROADCASTING);
break;
case CLUSTERING:
consumer.setMessageModel(org.apache.rocketmq.common.protocol.heartbeat.MessageModel.CLUSTERING);
break;
default:
throw new IllegalArgumentException("Property 'messageModel' was wrong.");
}
// 过滤方式
switch (selectorType) {
case TAG:
consumer.subscribe(topic, selectorExpression);
break;
case SQL92:
consumer.subscribe(topic, MessageSelector.bySql(selectorExpression));
break;
default:
throw new IllegalArgumentException("Property 'selectorType' was wrong.");
}
// 通过不同消费姿势,来注册不同的监听器
switch (consumeMode) {
case ORDERLY:
consumer.setMessageListener(new DefaultMessageListenerOrderly());
break;
case CONCURRENTLY:
consumer.setMessageListener(new DefaultMessageListenerConcurrently());
break;
default:
throw new IllegalArgumentException("Property 'consumeMode' was wrong.");
}
if (rocketMQListener instanceof RocketMQPushConsumerLifecycleListener) {
((RocketMQPushConsumerLifecycleListener) rocketMQListener).prepareStart(consumer);
}
}
}
DefaultMQPushConsumer
/**
* 1.推荐使用的消息消费方式
* 2.从技术上来说,这个push实现底层实际上是对pull方式的一个封装。从服务器定时拉取数据,来回调监听程序。
* 3.它是线程安全的
*/
public class DefaultMQPus