前言
rocketmq在Linux上搭建好了,现在说说rocketmq的默认producer与consumer方式。
1. rocketmq设计
我画了一张架构图
rocketmq的每条队列是顺序的,跟kafka的partition很相似;rocketmq默认通过随机正整数+1取模方式来选取队列的。rocketmq通过MessageQueueSelector保证消息的顺序发收,默认通过hash对队列数取模来实现消息发送到某条队列上,实现消息的顺序发送。
但如果任意一台broker异常,如连接中断,宕机,当Broker恢复前(主备切换或者broker重启等),因为队列总数改变,哈希取模路由的队列就会不同,就会造成短暂顺序不一致。
除了严格的顺序要求,一般使用默认方式就可以保证顺序,严格顺序会造成broker异常的短暂不可用。
rocketmq的每个组仅有一个节点消费消息。
下面说说默认实现方式
2. demo & pom
依赖client
<dependencies>
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.5.2</version>
</dependency>
</dependencies>
3. Producer
package com.feng.rocketmq.base;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import org.apache.rocketmq.remoting.exception.RemotingException;
import java.io.UnsupportedEncodingException;
public class Producer {
public static void main(String[] args) throws MQClientException, RemotingException, InterruptedException, MQBrokerException, UnsupportedEncodingException {
//instance
DefaultMQProducer producer = new DefaultMQProducer();
//group, for producer load balance
producer.setProducerGroup("demo-producer-group");
//namesrvAddr,cluster nameserver with ; spit
producer.setNamesrvAddr("127.0.0.1:9876");
//start
producer.start();
// send msg
int num = 20;
for (int i = 0; i < num; i++) {
//message,topic,tags is a mark in consumer,keys is a mark for message query
Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult result = producer.send(message, 1000);
System.out.println("send result is\t" + result);
}
//close, can use for rocket mq switch
producer.shutdown();
}
}
访问console
查看status
每个topic默认创建4个队列,即上面的queueId字段
4. consumer
consumer分为推和拉模式。推就是回调通知,监听器,来了消息就通知;拉是自己写代码定时拉取一定数量的消息。消费者需要自己写消费策略,需要考虑重复消费,消费业务考虑幂等性;需要处理不能消费的消息,防止消息堆积。
推:
缺点:消费过慢,不能处理的消息会造成消息堆积;无效的消息可能会反复推送
优点:消费及时,消息来了,监听器触发
拉:
缺点:需要大量的创建长连接,拉取的频率,拉取的数目需要严格考虑,是否使用长轮询
比如:消费者拉取失败,并不return,把连接挂起wait, 等待一段时间继续拉取,直到成功。
优点:不用考虑消息堆积;无效消息offset指向后即可处理
推的代码:
package com.feng.rocketmq.base;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import java.util.List;
public class PushConsumer {
public static void main(String[] args) throws MQClientException {
//instance
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
//group
consumer.setConsumerGroup("demo-consumer-group");
//setNamesrvAddr,cluster with ; spit
consumer.setNamesrvAddr("127.0.0.1:9876");
//consumer offset
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
//subscribe, the subExpression is tags in message send
//subscribe topic store in map
consumer.subscribe("demoTopic", "tags-1");
//can subscribe more
//consumer.subscribe("demoTopic2", "*");
//or use setSubscription, method is deprecated
//consumer.setSubscription();
//batch consumer max message limit
consumer.setConsumeMessageBatchMaxSize(1000);
//min thread
consumer.setConsumeThreadMin(10);
//listener
consumer.registerMessageListener((MessageListenerConcurrently) (list, consumeConcurrentlyContext) -> {
try {
for (MessageExt messageExt : list) {
if (messageExt.getReconsumeTimes() > 1){
continue;
}
String topic = messageExt.getTopic();
int queueId = messageExt.getQueueId();
String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body:" + message);
}
} catch (Exception e) {
//retry
return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
});
consumer.start();
//consumer.shutdown();
}
}
订阅topic存入map
消费成功
拉的代码:
package com.feng.rocketmq.base;
import org.apache.rocketmq.client.consumer.DefaultMQPullConsumer;
import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.PullResult;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.message.MessageQueue;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.apache.rocketmq.remoting.common.RemotingHelper;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class PullConsumer {
//record offset use key topic_queueId
private static final Map<String, Long> offsetMap = new ConcurrentHashMap<>();
public static void main(String[] args) throws MQClientException, InterruptedException {
//instance
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer();
//group
consumer.setConsumerGroup("demo-consumer-group");
//namesrv
consumer.setNamesrvAddr("127.0.0.1:9876");
//cluster model
consumer.setMessageModel(MessageModel.CLUSTERING);
consumer.start();
//pull all mq with cluster or broadcast
Set<MessageQueue> mqs = null;
switch (consumer.getMessageModel()) {
case BROADCASTING:
mqs = consumer.fetchSubscribeMessageQueues("demoTopic");
break;
case CLUSTERING:
//set topic load balance
consumer.registerMessageQueueListener("demoTopic", null);
//cluster, each node get some mq
mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
if (mqs == null || mqs.isEmpty()) {
while (mqs == null || mqs.isEmpty()){
Thread.sleep(1000l);
mqs = consumer.fetchMessageQueuesInBalance("demoTopic");
}
}
break;
}
ScheduledThreadPoolExecutor scheduledThreadPoolExecutor = new ScheduledThreadPoolExecutor(4);
for (MessageQueue mq : mqs) {
scheduledThreadPoolExecutor.scheduleAtFixedRate(() -> {
try {
PullResult pullResult = consumer.pullBlockIfNotFound(mq, "tags-1", offerMQOffset(mq, consumer.fetchConsumeOffset(mq, true)), 10);
System.out.println(pullResult);
switch (pullResult.getPullStatus()) {
case FOUND:
List<MessageExt> messageExtList = pullResult.getMsgFoundList();
//can async deal
for (MessageExt m : messageExtList) {
System.out.println(new String(m.getBody()));
}
saveMQOffset(mq, pullResult.getNextBeginOffset());
break;
case NO_MATCHED_MSG:
case NO_NEW_MSG:
case OFFSET_ILLEGAL:
}
} catch (Exception e) {
e.printStackTrace();
}
}, 5l, 5l, TimeUnit.SECONDS);
System.out.println("topic & queueId : \t"+mq.getTopic()+"_"+mq.getQueueId());
}
//consumer.shutdown();
}
private static void saveMQOffset(MessageQueue mq, long offset) {
offsetMap.put(mq.getTopic() + "_" + mq.getQueueId(), offset);
}
private static long offerMQOffset(MessageQueue mq, long defaultOffset) {
Long offset = offsetMap.get(mq.getTopic() + "_" + mq.getQueueId());
if (defaultOffset < 0) {
defaultOffset = 0;
}
return offset == null ? defaultOffset : offset;
}
}
这里非常关键
consumer.registerMessageQueueListener("demoTopic", null);
源码可以看到将topic注册进loadbalance列表中了,笔者最开始各种load balance拉取都拉取不到message queue;原因是没有负责均衡。
另外在拉取的messageExtList,一般是要异步线程池处理的,提高并发量。
rocketmq官方提供
MQPullConsumerScheduleService
用于支持定时pull 消息,相当于封装好了拉取逻辑,推荐使用,不用自己写一堆代码,下一章写一个demo,分析原理。
5. spring-boot集成rocketmq
在spring-boot框架中集成rocketmq非常简单,将上面的代码转变成springboot方式即可
5.1 配置属性
package com.feng.springboot.rocketmq.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
@ConfigurationProperties(prefix = "org.rocketmq")
@Data
public class RocketMQProperties {
private String nameSrvs;
private String producerGroup;
}
application.properties文件,yml同理
org.rocketmq.nameSrvs=127.0.0.1:9876
org.rocketmq.producerGroup=producer_demo_group
5.2 配置bean
package com.feng.springboot.rocketmq.config;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.MQProducer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@EnableConfigurationProperties(RocketMQProperties.class)
@Configuration
public class RocketMQInitConfig {
@Autowired
private RocketMQProperties rocketMQProperties;
@Bean
public MQProducer getMQProducer() throws MQClientException {
DefaultMQProducer mqProducer = new DefaultMQProducer();
mqProducer.setNamesrvAddr(rocketMQProperties.getNameSrvs());
mqProducer.setProducerGroup(rocketMQProperties.getProducerGroup());
mqProducer.setSendMsgTimeout(1000);
mqProducer.setVipChannelEnabled(false);
//set other properties
//............
mqProducer.start();
return mqProducer;
}
}
好了,集成进来了,可以使用了
@SpringBootApplication
@RestController
public class RocketMQMain {
public static void main(String[] args) {
SpringApplication.run(RocketMQMain.class, args);
}
@Autowired
private MQProducer producer;
@RequestMapping(value = "messages", method = RequestMethod.GET)
public Map<String, String> sendMsg() throws InterruptedException, RemotingException, MQClientException, MQBrokerException, UnsupportedEncodingException {
Map<String, String> map = new HashMap<>(2);
// send msg
int num = 20;
for (int i = 0; i < num; i++) {
//message,topic,tags is a mark in consumer,keys is a mark for message query
Message message = new Message("demoTopic","tags-1", "instanceKeys", ("I`m a " + i + " rocket mq msg!").getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult result = producer.send(message, 1000);
System.out.println("send result is\t" + result);
}
map.put("isSend", "OK");
return map;
}
}
运行后访问http://localhost:8080/messages
控制台
说明集成成功,同理集成consumer
@Autowired
private RocketMQDemoListener rocketMQDemoListener;
@Bean
public MQConsumer getMQConsumer() throws MQClientException {
//instance
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer();
//group
consumer.setConsumerGroup(rocketMQProperties.getConsumerGroup());
//setNamesrvAddr,cluster with ; spit
consumer.setNamesrvAddr(rocketMQProperties.getNameSrvs());
//consumer offset
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
//subscribe, the subExpression is tags in message send
//subscribe topic store in map
consumer.subscribe("demoTopic", "tags-1");
//can subscribe more
//consumer.subscribe("demoTopic2", "*");
//or use setSubscription, method is deprecated
//consumer.setSubscription();
//batch consumer max message limit
consumer.setConsumeMessageBatchMaxSize(1000);
//min thread
consumer.setConsumeThreadMin(10);
consumer.registerMessageListener(rocketMQDemoListener);
consumer.start();
return consumer;
}
listener逻辑,在自定义的listener里面可以做一个代理层,埋下一个钩子,方便以后业务处理。
@Component
public class RocketMQDemoListener implements MessageListenerConcurrently {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
try {
for (MessageExt messageExt : msgs) {
if (messageExt.getReconsumeTimes() > 1){
continue;
}
String topic = messageExt.getTopic();
int queueId = messageExt.getQueueId();
String message = new String(messageExt.getBody(), RemotingHelper.DEFAULT_CHARSET);
System.out.println("the topic: " + topic + "\tqueueId:" + queueId + "\t body:" + message);
}
} catch (Exception e) {
//retry
return ConsumeConcurrentlyStatus.RECONSUME_LATER;
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
}
将RocketMQInitConfig加入springboot的EnableAutoConfiguration即可做成rocketmq插件starter
在src/main/resource
目录下创建META-INF
目录,然后创建文件spring.factories
:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=自己写的config类(全路径)
总结
rocketmq 其实生产与消费很简单,复杂的地方在group的loadbalance,消息在broker上的分配。rocketmq有严格执行顺序的生产消费方式,并支持分布式事务,由消息传递分布式各节点的事务情况,生产者根据结果中心化处理事务情况。