文章目录
消息如何保证100%投递成功
什么是生产端的可靠性投递
- 保障消息的成功发出
- 保障MQ节点的成功接收
- 发送端收到MQ节点(Broker)确认应答
- 完善的消息进行补偿机制
生产端-可靠性传递
常见解决方案:
消息落库,对消息状态进行打标
BIZ DB:订单数据库(或其他具体业务)
MSG DB:消息数据库
第1步:将订单入库,创建一条MSG(状态为0) 入MSG DB库
第2步:将消息发出去
第3步:监听消息应答(来自Broker)
第4步:修改消息的状态为1(成功)
第5步:分布式定时任务抓取状态为0的消息
第6步:将状态为0的消息重发
第7步:如果尝试了3次(可按实际情况修改)以上则将状态置为2(消息投递失败状态)
这种方案,需要入两次库,在高并发的场景下性能可能不是那么好
消息的延迟投递,做二次确认,回调检查
第二种方案可以避免消息对象与业务对象同时入库
Upstream service:上游服务,可能为生产端
Downstream service:下游服务,可能为消费端
MQ Broker:可能为集群
Callback service:回调服务,监听confirm消息
第1步:首先业务数据落库,成功才后第一次消息发送
第2步:紧着着发送第2条消息(可以用于寻找第1条消息),用于延迟(可能2,3分钟后才发送)消息投递检查
第3步:Broker端收到消息后,消费端进行消息处理
第4步:处理成功后,发送confirm消息
第5步:收到confirm消息后,将消息进行持久化存储
第6步:收到了delay消息,检查DB数据库,若对应的第1条消息已处理完成,则不做任何事情;若收到了delay消息,检查DB数据库,发现对应的第1条消息处理失败(或无记录),则发送重传命令到上游服务,循环第1步
消息幂等性
幂等性
比如我们执行一条更新库存的SQL语句(乐观锁):
update T_REPS set count = count - 1,version = version + 1 where version = 1
第一步,查出version
第二步,通过这个version进行更新
可以进行幂等性保障
所以,啥是幂等性呢?
执行某个操作,无论执行多少次,结果都是一致的,就说具有幂等性。
如何避免重复消费
在海量订单产生的业务高峰期,如何避免消息的重复消费?
- 消费端实现幂等性,然后永远不会消费多次,即使收到多条一样的消息
主流解决方案
消费端实现幂等性的主流解决方案有以下两种
唯一ID +指纹码 机制
利用数据库主键去重
指纹码:可能是业务规则,时间戳+具体银行范围的唯一信息码,能保障这次操作的绝对唯一
比如select count(1) from T_ORDER where id = <唯一ID+指纹码>
将唯一ID+指纹码设成主键,如果上面SQL返回1,说明已经操作了,则不需要再次操作;否则才去执行操作
优点: 实现简单
缺点:高并发下有数据库写入的性能瓶颈(解决方案:通过ID进行分库分表进行算法路由)
利用Redis的原子性实现
- 通过setnx等命令
SET 订单号 时间戳 过期时间
SET 1893505609317740 1466849127 EX 300 NX
利用Redis进行幂等,需要考虑的问题:
- 如果要进行数据落库,关键解决的问题是数据库和缓存如何做到数据一致性。
- 如果不落库,那么都存在缓存中,如何设置定时同步的策略(同步是指将数据存储到数据库中,不落库指的是暂时不落库,不可能永远不落库)
投递消息机制
Confirm确认消息
- 消息确认,是指生产者消息投递后,如果Broker收到消息,则会给生产者一个应答
- 生产者进行接收应答,用来确定这条消息是否正常的发送到Broker,这种方式也是消息的可靠性投递的核心保障
上图是确认机制流程图
生成者发送消息与监听confirm是异步的。
如何实现Confirm确认消息?
- 在channel上开启确认模式:
channel.confirmSelect()
- 在channel上添加监听:
addConfirmListener
,监听成功和失败的返回结果,根据具体返回结果对消息进行重发或日志记录等
Producer:
public class Producer {
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
//2.指定消息确认模式
channel.confirmSelect();
String exchangeName = "test_confirm_exchange";
String routingKey = "confirm.save";
//3. 通过Channel发送数据
String message = "Hello from Producer";
channel.basicPublish(exchangeName,routingKey,null,message.getBytes());
//4. 添加一个确认监听
channel.addConfirmListener(new ConfirmListener() {
@Override
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
//成功的情况 deliveryTag:消息的唯一标签;
System.out.println("——get ack——");
}
@Override
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
//失败的情况
System.out.println("——have no ack——");
}
});
// 关闭掉就没confirm了
// CloseTool.closeElegantly(channel,connection);
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchangeName = "test_confirm_exchange";
String routingKey = "confirm.save";
String queueName = "test_confirm_queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchangeName,"topic",true);
//3. 声明一个队列
channel.queueDeclare(queueName,true,false,false,null);
//4. 绑定
channel.queueBind(queueName,exchangeName,routingKey);
//5. 创建消费者
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
//6. 设置Channel
channel.basicConsume(queueName,true,queueingConsumer);
//7. 获取消息
while (true) {
//nextDelivery 会阻塞直到有消息过来
QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println("收到:" + message);
}
}
}
return返回消息
- Return Listener用于处理一些不可路由消息
- 生产者指定Exchange和RoutingKey,将消息投递到某个队列,然后消费者监听队列,进行消息处理
- 但在某些情况下,在发送消息时,若当前的exchange不存在或指定的路由key路由失败,这时,如果需要监听这种不可达的消息,则要使用return listener
return 消息机制
- 在基础API中有一个关键的配置项:
Mandatory
: 若为true
,则监听器会接收到路由不可达的消息,然后进行后粗处理 ;若为false
,则broker端自动删除该消息
发送端发送了一条消息,但是没有发现Exchange,则可通过return listener监听这些消息
producer:
public class Producer {
public static final String MQ_HOST = "192.168.222.101";
public static final String MQ_VHOST = "/";
public static final int MQ_PORT = 5672;
public static void main(String[] args) throws IOException, TimeoutException {
//1. 创建一个ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(MQ_HOST);//配置host
connectionFactory.setPort(MQ_PORT);//配置port
connectionFactory.setVirtualHost(MQ_VHOST);//配置vHost
//2. 通过连接工厂创建连接
Connection connection = connectionFactory.newConnection();
//3. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_return_exchange";
String routingKey = "return.save";
String routingKeyError = "abc.save";
//4. 通过Channel发送数据
String message = "Hello Return Message";
channel.addReturnListener((replyCode, replyText, exchange1, routingKey1, properties, body) -> {
System.out.println("——handle return——");
System.out.println("replyCode:" + replyCode);
System.out.println("replyText:" + replyText);
System.out.println("exchange1:" + exchange1);
System.out.println("routingKey1:" + routingKey1);
System.out.println("properties:" + properties);
System.out.println("body:" + new String(body));
});
//mandatory : true
//channel.basicPublish(exchange,routingKey,true,null,message.getBytes());
channel.basicPublish(exchange,routingKeyError,true,null,message.getBytes());
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_return_exchange";
String routingKey = "return.#";
String queueName = "test_return_queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchange,"topic",true,false,null);
//3. 声明一个队列
channel.queueDeclare(queueName,true,false,false,null);
//4. 绑定
channel.queueBind(queueName,exchange,routingKey);
//5. 创建消费者
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
//6. 设置Channel
channel.basicConsume(queueName,true,queueingConsumer);
//7. 获取消息
while (true) {
//nextDelivery 会阻塞直到有消息过来
QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println("收到:" + message);
}
}
}
当生产端执行channel.basicPublish(exchange,routingKey,true,null,message.getBytes());
消息能发送成功,也可以从消费端看到打印
当执行channel.basicPublish(exchange,routingKeyError,true,null,message.getBytes());
消息发送失败,因为路由失败了嘛,生产端能看到如下打印:
——handle return——
replyCode:312
replyText:NO_ROUTE
exchange1:test_return_exchange
routingKey1:abc.save
properties:#contentHeader<basic>(content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null)
body:Hello Return Message
若生产端将mandatory设为false,则ReturnListener不会进行回调
消费端自定义监听
- 如何更优雅的实现消息监听代码
- 需要继承
DefaultConsumer
,重写handleDelivery()
方法
produer:
public class Producer {
public static final String MQ_HOST = "192.168.222.101";
public static final String MQ_VHOST = "/";
public static final int MQ_PORT = 5672;
public static void main(String[] args) throws IOException, TimeoutException {
//1. 创建一个ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(MQ_HOST);//配置host
connectionFactory.setPort(MQ_PORT);//配置port
connectionFactory.setVirtualHost(MQ_VHOST);//配置vHost
//2. 通过连接工厂创建连接
Connection connection = connectionFactory.newConnection();
//3. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_consumer_exchange";
String routingKey = "consumer.save";
//4. 通过Channel发送数据
String message = "Hello Consumer Message";
for (int i = 0; i < 5; i++) {
channel.basicPublish(exchange,routingKey,null,message.getBytes());
}
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_consumer_exchange";
String routingKey = "consumer.save";
String queueName = "test_consumer_queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchange,"topic",true,false,null);
//3. 声明一个队列
channel.queueDeclare(queueName,true,false,false,null);
//4. 绑定
channel.queueBind(queueName,exchange,routingKey);
//5. 创建消费者
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
//6. 设置Channel
channel.basicConsume(queueName,true,new MyConsumer(channel));
/* //7. 获取消息
之前的方式,很ugly
while (true) {
//nextDelivery 会阻塞直到有消息过来
QueueingConsumer.Delivery delivery = queueingConsumer.nextDelivery();
String message = new String(delivery.getBody());
System.out.println("收到:" + message);
}*/
}
private static class MyConsumer extends DefaultConsumer {
public MyConsumer(Channel channel) {
super(channel);
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("——consume message——");
System.out.println("consumerTag:"+consumerTag);
System.out.println("envelope:"+envelope);
System.out.println("properties:"+properties);
System.out.println("body:"+new String(body));
}
}
}
打印如下:
——consume message——
consumerTag:amq.ctag-DLKq_dy8aYspCTUBrnHTew
envelope:Envelope(deliveryTag=1, redeliver=false, exchange=test_consumer_exchange, routingKey=consumer.save)
properties:#contentHeader<basic>(content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null)
body:Hello Consumer Message
——consume message——
consumerTag:amq.ctag-DLKq_dy8aYspCTUBrnHTew
envelope:Envelope(deliveryTag=2, redeliver=false, exchange=test_consumer_exchange, routingKey=consumer.save)
properties:#contentHeader<basic>(content-type=null, content-encoding=null, headers=null, delivery-mode=null, priority=null, correlation-id=null, reply-to=null, expiration=null, message-id=null, timestamp=null, type=null, user-id=null, app-id=null, cluster-id=null)
body:Hello Consumer Message
...
消息的限流
什么是消费端的限流:
- 假设,MQ服务器上有上万条未处理的消息,随便打开一个消费者客户端,会出现下面情况:
- 巨量的消息瞬间全部推送过来,但是单个客户端无法同时处理这么多数据,从而导致服务器崩溃
解决方案:
- RabbitMQ提供了一种qos(服务质量保证)功能,即在非自动确认消息的前提下(设置
autoAck
为false
),若一定数量的消息(通过基于consume或channel设置Qos的值)未被确认前,不消费新的消息 void BasicQos(unit prefetchSize,unshort prefetchCount,bool global)
- prefetchSize:0 消费的单挑消息的大小限制,0代表不限制
- prefetchCount:不要同时给一个消费者推送多于N个消息,即一旦有N个消息还没有ack,则该consumer将block掉,直到有消息ack
- global:
true\false
true
:上面设置应用于channel
级别;false
:应用于consumer
级别
producer:
public class Producer {
public static final String MQ_HOST = "192.168.222.101";
public static final String MQ_VHOST = "/";
public static final int MQ_PORT = 5672;
public static void main(String[] args) throws IOException, TimeoutException {
//1. 创建一个ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(MQ_HOST);//配置host
connectionFactory.setPort(MQ_PORT);//配置port
connectionFactory.setVirtualHost(MQ_VHOST);//配置vHost
//2. 通过连接工厂创建连接
Connection connection = connectionFactory.newConnection();
//3. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_qos_exchange";
String routingKey = "qos.save";
//4. 通过Channel发送数据
String message = "Hello Qos Message";
for (int i = 0; i < 5; i++) {
channel.basicPublish(exchange,routingKey,null,message.getBytes());
}
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_qos_exchange";
String routingKey = "qos.#";
String queueName = "test_qos_queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchange,"topic",true,false,null);
//3. 声明一个队列
channel.queueDeclare(queueName,true,false,false,null);
//4. 绑定
channel.queueBind(queueName,exchange,routingKey);
//5. 创建消费者
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
//6.限流方式 每次只推3条
channel.basicQos(0,3,false);
//7. 设置Channel autoAck一定要设置为false,才能做限流
channel.basicConsume(queueName,false,new MyConsumer(channel));
}
private static class MyConsumer extends DefaultConsumer {
private Channel channel;
public MyConsumer(Channel channel) {
super(channel);
this.channel = channel;
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("——consume message——");
System.out.println("consumerTag:"+consumerTag);
System.out.println("envelope:"+envelope);
System.out.println("properties:"+properties);
System.out.println("body:"+new String(body));
// 手动签收
channel.basicAck(envelope.getDeliveryTag(),false);
}
}
}
消费端的ACK与重回队列
消费端可以进行手工ACK和NACK(不确认,表示失败)
- 消费端进行消费时,如果由于业务异常,可以进行日志记录,然后进行补偿
- 如果由于服务器宕机等严重问题,需要手工进行ACK保障消费端消费成功
消费端的重回队列:
- 对没有处理成功的消息,把消息重新传递给Broker
- 一般在实际应用中,会关闭重回队列
producer:
public class Producer {
public static final String MQ_HOST = "192.168.222.101";
public static final String MQ_VHOST = "/";
public static final int MQ_PORT = 5672;
public static void main(String[] args) throws IOException, TimeoutException {
//1. 创建一个ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(MQ_HOST);//配置host
connectionFactory.setPort(MQ_PORT);//配置port
connectionFactory.setVirtualHost(MQ_VHOST);//配置vHost
//2. 通过连接工厂创建连接
Connection connection = connectionFactory.newConnection();
//3. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_ack_exchange";
String routingKey = "ack.save";
//4. 通过Channel发送数据
String message = "Hello Ack Message";
for (int i = 0; i < 5; i++) {
Map<String,Object> headers = new HashMap<>();
headers.put("num",i);
AMQP.BasicProperties properties = new AMQP.BasicProperties().builder()
.deliveryMode(2) //持久化
.contentEncoding("UTF-8")
.headers(headers)
.build();
channel.basicPublish(exchange,routingKey,properties,message.getBytes());
}
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_ack_exchange";
String routingKey = "ack.#";
String queueName = "test_ack_queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchange,"topic",true,false,null);
//3. 声明一个队列
channel.queueDeclare(queueName,true,false,false,null);
//4. 绑定
channel.queueBind(queueName,exchange,routingKey);
//5. autoAck = false
channel.basicConsume(queueName,false,new MyConsumer(channel));
}
private static class MyConsumer extends DefaultConsumer {
private Channel channel;
public MyConsumer(Channel channel) {
super(channel);
this.channel = channel;
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("——consume message——");
System.out.println("body:"+new String(body));
System.out.println("num:" + properties.getHeaders().get("num"));
if ((Integer)properties.getHeaders().get("num") == 0 ) {
//requeue:true表示重新入队,重传
channel.basicNack(envelope.getDeliveryTag(),false,true);
} else {
channel.basicAck(envelope.getDeliveryTag(),false);
}
}
}
}
日志:
...
——consume message——
body:Hello Ack Message
num:0
——consume message——
body:Hello Ack Message
num:0
——consume message——
body:Hello Ack Message
num:0
...
num为0的消息会一直回到MQ队列的最尾端,然后一直循环打印,因为一直无法消费
TTL消息
- 生存时间(Time to Live,TTL),指的是消息的生成时间
- RabbitMQ支持消息的过期时间,在消息发送时可以进行指定
- 还支持队列的过期时间,从消息入队列开始计时,只要超过了队列的超时时间,那么消息会自动的清除
消息TTL:
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.deliveryMode(2) //2:持久化投递;1:非持久化(未消费的消息重启后就没了)
.contentEncoding("UTF-8")
.expiration("5000")//5s 设置消息的TTL
.headers(headers)
.build();
String message = "Hello";
channel.basicPublish("","testQueue",properties,message.getBytes());
}
队列TTL:
Map<String, Object> args = new HashMap<>();
args.put("x-expires", 1800000); //30分钟
channel.queueDeclare("myqueue", false, false, false, args);
死信队列
- 死信队列(Dead-Letter-Exchange,DLX)
- 利用DLX,当消息在队列中变成死信(dead message:没有消费者去消费)之后,它能被重新publish到另一个Exchange,这个Exchange就是死信队列
- DLX是一个正常的Exchange,和一般的Exchange没有区别,它能在任何的队列上被指定
- 当这个队列中有死信时,RabbitMQ就会自动的将这个消息重新发布到设置的Exchange上,进而被路由到另一个队列
- 可以监听这个队列中的消息并做相应的处理
消息变成死信的情况
- 消息被拒绝(
basic.rejcet/basic.nack
)且requeue=false
- 消息TTL过期
- 队列达到最大长度
死信队列设置
- 首先要设置死信队列的exchange和queue,然后进行绑定:
- Exchange:dlx.exchange
- Queue:dlx.queue
- RoutingKey:#
- 然后进行正常声明交换机、队列、绑定,只不过需要在队列上加上一个参数:
arguments.put("x-dead-letter-exchange","dlx.exchange);
- 这样消息在过期、requeue、队列达到最大长度时,消息就可以直接路由到死信队列
producer:
public class Producer {
public static final String MQ_HOST = "192.168.222.101";
public static final String MQ_VHOST = "/";
public static final int MQ_PORT = 5672;
public static void main(String[] args) throws IOException, TimeoutException {
//1. 创建一个ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost(MQ_HOST);//配置host
connectionFactory.setPort(MQ_PORT);//配置port
connectionFactory.setVirtualHost(MQ_VHOST);//配置vHost
//2. 通过连接工厂创建连接
Connection connection = connectionFactory.newConnection();
//3. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_dlx_exchange";
String routingKey = "dlx.save";
//4. 通过Channel发送数据
String message = "Hello DLX Message";
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.deliveryMode(2) //2:持久化投递;1:非持久化(未消费的消息重启后就没了)
.contentEncoding("UTF-8")
.expiration("5000")//5s后如果没有消费端消费,会变成死信
.build();
for (int i = 0; i < 1; i++) {
channel.basicPublish(exchange,routingKey,properties,message.getBytes());
}
}
}
consumer:
public class Consumer {
public static void main(String[] args) throws Exception {
Connection connection = ConnectionUtil.getConn();
//1. 通过connection创建一个Channel
Channel channel = connection.createChannel();
String exchange = "test_dlx_exchange";
String routingKey = "dlx.#";
String queueName = "test_dlx_queue";
String dlxExchange = "dlx.exchange";
String dlxQueue = "dlx.queue";
//2. 声明一个exchange
channel.exchangeDeclare(exchange,"topic",true,false,null);
Map<String,Object> arguments = new HashMap<>();
//路由失败,重发到dlx.exchange
arguments.put("x-dead-letter-exchange",dlxExchange);
/**
* 声明正常队列
* arguments要设置到声明队列上
*/
channel.queueDeclare(queueName,true,false,false,arguments);
channel.queueBind(queueName,exchange,routingKey);
//进行死信队列的声明
channel.exchangeDeclare(dlxExchange,"topic",true,false,null);
channel.queueDeclare(dlxQueue,true,false,false,null);
channel.queueBind(dlxQueue,dlxExchange,"#");
}
private static class MyConsumer extends DefaultConsumer {
public MyConsumer(Channel channel) {
super(channel);
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("——consume message——");
System.out.println("body:"+new String(body));
}
}
}
先启动Consumer,进行生成队列等操作
然后
查看Exchanges:
查看Queues:
接着,为了让消息变成死信,停止Consumer
最后,启动Producer
5秒后,消息没被消费,然后就进去了死信队列(注意死信队列中初始值是1)