Rocket顺序消息
本篇基于RocketMQ集群开发,如需了解集群的搭建,可参考之前的文章RocketMQ集群安装。
摘录RocketMQ官方示例中顺序消息的解释信息:
1.顺序消息指按照消息发送的顺序(FIFO)消费。RocketMQ支持分区有序和全局有序。
2.在默认的情况下消息发送会采取Round Robin轮询方式把消息发送到不同的queue(分区队列);而消费消息的时候从多个queue上拉取消息,这种情况发送和消费是不能保证顺序。但是如果控制发送的顺序消息只依次发送到同一个queue中,消费的时候只从这个queue上依次拉取,则就保证了顺序。当发送和消费参与的queue只有一个,则是全局有序;如果多个queue参与,则为分区有序,即相对每个queue,消息都是有序的。
顺序队列的关键就是让同一个订单的消息,按顺序放入同一个队列。
下面用订单进行分区有序的示例。一个订单的顺序流程是:创建、付款、推送、完成。订单号相同的消息会被先后发送到同一个队列中,消费时,同一个OrderId获取到的肯定是同一个队列。
Producer生产者
生产者使用send(Message msg, MessageQueueSelector selector, Object arg)
方法,控制订单消息分区有序需要MessageQueueSelector对订单号相同的返回同一个MessageQueue消息队列。
public class OrderProducer {
public static void main(String[] args) throws Exception {
// 1.指定生产者组
DefaultMQProducer defaultMQProducer = new DefaultMQProducer(MyRocketMqConstant.OrderLearn.ORDER_LEARN_PRODUCER_GROUP);
// 2.关联注册中心
defaultMQProducer.setNamesrvAddr(MyRocketMqConstant.NAME_SRV);
// 3.启动生产者
defaultMQProducer.start();
String[] tags = {MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_A
, MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_B
, MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_C};
// 模拟订单数据
List<OrderStep> orderStepList = new OrderProducer().buildOrders();
Date date = new Date();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
String dateStr = sdf.format(date);
for (int i = 0; i < 10; i++) {
// 构造消息数据
String body = orderStepList.get(i) + "Hello RocketMQ" + dateStr;
Message message = new Message(MyRocketMqConstant.OrderLearn.ORDER_LEARN_TOPIC
, tags[i % tags.length]
, "KEY" + i, body.getBytes(StandardCharsets.UTF_8));
SendResult sendResult = defaultMQProducer.send(message, new MessageQueueSelector() {
// 消息队列对象
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
// arg来自于send(Message msg, MessageQueueSelector selector, Object arg)的第三个参数arg
// 同一订单号使用同一消息队列
Long id = (Long) arg;
long index = id % mqs.size();
return mqs.get((int) index);
}
}, orderStepList.get(i).getOrderId());
System.out.println(String.format("SendResult status:%s,queueId:%d,body:%s",
sendResult.getSendStatus(),
sendResult.getMessageQueue().getQueueId(),
body));
}
defaultMQProducer.shutdown();
}
/**
* 订单的步骤
*/
private static class OrderStep {
private long orderId;
private String desc;
public long getOrderId() {
return orderId;
}
public void setOrderId(long orderId) {
this.orderId = orderId;
}
public String getDesc() {
return desc;
}
public void setDesc(String desc) {
this.desc = desc;
}
@Override
public String toString() {
return "OrderStep{" +
"orderId=" + orderId +
", desc='" + desc + '\'' +
'}';
}
}
/**
* 生成模拟订单数据
* 保证订单的生成顺序是:创建->付款->推送->完成
*/
private List<OrderStep> buildOrders() {
List<OrderStep> orderList = new ArrayList<OrderStep>();
OrderStep orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("创建");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("付款");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111065L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("推送");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103117235L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
orderDemo = new OrderStep();
orderDemo.setOrderId(15103111039L);
orderDemo.setDesc("完成");
orderList.add(orderDemo);
return orderList;
}
}
Consumer消费者
public class OrderConsumer {
public static void main(String[] args) throws Exception {
// 1.指定消费者组
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(MyRocketMqConstant.OrderLearn.ORDER_LEARN_CONSUMER_GROUP);
// 2.关联注册中心
consumer.setNamesrvAddr(MyRocketMqConstant.NAME_SRV);
// 3.设置Consumer第一次启动是从队列头部开始消费还是从队列尾部开始消费
// 如果非第一次启动,那么按照上次消费的位置继续消费
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
String[] tags = {MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_A
, MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_B
, MyRocketMqConstant.OrderLearn.ORDER_LEARN_TAG_C};
String join = String.join(" || ", tags);
consumer.subscribe(MyRocketMqConstant.OrderLearn.ORDER_LEARN_TOPIC, join);
consumer.registerMessageListener(new MessageListenerOrderly() {
Random random = new Random();
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
for (MessageExt msg : msgs) {
// 每个queue不一定有唯一的consumer线程消费,订单对每个queue有序
System.out.println("[consumerThread="+Thread.currentThread().getName()+"] [queueId="+msg.getQueueId()+"] content"+new String(msg.getBody()));
}
try {
// 模拟业务逻辑处理...
TimeUnit.SECONDS.sleep(random.nextInt(10));
} catch (InterruptedException e) {
e.printStackTrace();
}
return ConsumeOrderlyStatus.SUCCESS;
}
});
consumer.start();
}
}
tips:RocketMQ源码中示例说:每个queue有唯一的consumer线程消费,订单对每个queue有序,实践证明这个描述不正确。正确的是:每个queue不一定有唯一的consumer线程消费,订单对每个queue有序。
测试结果
总共测试了三次,输出结果如下(可以把输出内容放入notepad++中,双击订单号查看同订单号的输出顺序)。
发现消费者的输出顺序不一定会一致,但是针对同一个订单的输出顺序是一定相同的。同时消费者消费时,同一个订单号可能有不同的线程消费。
生产者输出
生产者
//队列3
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103111039, desc='创建'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103111039, desc='付款'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103117235, desc='创建'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103117235, desc='付款'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103111039, desc='推送'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103117235, desc='完成'}
SendResult status:SEND_OK,queueId:3,body:OrderStep{orderId=15103111039, desc='完成'}
//队列1
SendResult status:SEND_OK,queueId:1,body:OrderStep{orderId=15103111065, desc='创建'}
SendResult status:SEND_OK,queueId:1,body:OrderStep{orderId=15103111065, desc='付款'}
SendResult status:SEND_OK,queueId:1,body:OrderStep{orderId=15103111065, desc='完成'}
消费者输出
消费者
//队列3
[consumerThread=ConsumeMessageThread_1] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='创建'}]
[consumerThread=ConsumeMessageThread_3] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='创建'}]
[consumerThread=ConsumeMessageThread_1] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='付款'}]
[consumerThread=ConsumeMessageThread_1] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='推送'}]
[consumerThread=ConsumeMessageThread_1] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='完成'}]
[consumerThread=ConsumeMessageThread_3] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='付款'}]
[consumerThread=ConsumeMessageThread_3] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='完成'}]
//队列1
[consumerThread=ConsumeMessageThread_2] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='创建'}]
[consumerThread=ConsumeMessageThread_2] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='付款'}]
[consumerThread=ConsumeMessageThread_2] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='完成'}]
//==================================================测试分隔线===================================================================================
//队列3
[consumerThread=ConsumeMessageThread_4] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='创建'}]
[consumerThread=ConsumeMessageThread_6] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='创建'}]
[consumerThread=ConsumeMessageThread_4] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='付款'}]
[consumerThread=ConsumeMessageThread_6] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='付款'}]
[consumerThread=ConsumeMessageThread_6] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='完成'}]
[consumerThread=ConsumeMessageThread_4] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='推送'}]
[consumerThread=ConsumeMessageThread_4] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='完成'}]
//队列1
[consumerThread=ConsumeMessageThread_5] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='创建'}]
[consumerThread=ConsumeMessageThread_5] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='付款'}]
[consumerThread=ConsumeMessageThread_5] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='完成'}]
//==================================================测试分隔线===================================================================================
//队列3
[consumerThread=ConsumeMessageThread_7] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='创建'}]
[consumerThread=ConsumeMessageThread_9] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='创建'}]
[consumerThread=ConsumeMessageThread_10] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='付款'}]
[consumerThread=ConsumeMessageThread_10] [broker=broker-a/192.168.15.15:10911] [queueId=3] [content=OrderStep{orderId=15103117235, desc='完成'}]
[consumerThread=ConsumeMessageThread_7] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='付款'}]
[consumerThread=ConsumeMessageThread_7] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='推送'}]
[consumerThread=ConsumeMessageThread_7] [broker=broker-b/192.168.15.16:10911] [queueId=3] [content=OrderStep{orderId=15103111039, desc='完成'}]
//队列1
[consumerThread=ConsumeMessageThread_8] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='创建'}]
[consumerThread=ConsumeMessageThread_8] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='付款'}]
[consumerThread=ConsumeMessageThread_8] [broker=broker-a/192.168.15.15:10911] [queueId=1] [content=OrderStep{orderId=15103111065, desc='完成'}]
总结
消费时同一订单号的消息顺序是如何保证和生产者生成订单相同的?需要关注MessageQueueSelector,它的返回值就是消息所关联的队列。
MessageQueueSelector
创建的匿名内部类如下:
new MessageQueueSelector() {
// 消息队列对象
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
// arg来自于send(Message msg, MessageQueueSelector selector, Object arg)的第三个参数arg
// 同一订单号使用同一消息队列
Long id = (Long) arg;
long index = id % mqs.size();
return mqs.get((int) index);
}
测试环境使用了集群,2个broker-a,2个broker-b;创建MessageQueueSelector对象时,参数List<MessageQueue> mqs
内容如下,有8条记录。Long id = (Long) arg;
获取到send方法传入的订单id,long index = id % mqs.size();
取余会使同一订单id的不同消息关联同一个消息队列。
根据上面的消费者输出测试结果,三次输出的消费者信息顺序虽然不完全一致,但是同一个订单id的顺序一定是一致的,因为同一订单号的消息关联了同一个消息队列,这就是RocketMQ的分区顺序。
[
{
"brokerName": "broker-a","queueId": 0,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 1,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 2,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 3,"topic": "order-topic"
},
{
"brokerName": "broker-b","queueId": 0,"topic": "order-topic"
},
{
"brokerName": "broker-b","queueId": 1,"topic": "order-topic"
},
{
"brokerName": "broker-b","queueId": 2,"topic": "order-topic"
},
{
"brokerName": "broker-b","queueId": 3,"topic": "order-topic"
}
]
关闭slave02上的broker(启动了broker-b主节点和broker-a的从节点),启动生产者获取到的mqs对象内容如下:
[
{
"brokerName": "broker-a","queueId": 0,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 1,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 2,"topic": "order-topic"
},
{
"brokerName": "broker-a","queueId": 3,"topic": "order-topic"
}
]