本章导航
- 消息如何保障100%投递成功
- 幂等性概念详解
- 在海量订单产生的业务高峰期，如何避免消息的重复消费等问题
- Confirm确认消息、Return返回消息
- 自定义消费者
- 消息的ACK和重回队列
- 消息的限流
- TTL消息
- 死信队列
3.1消息如何保障100%投递成功
什么是生产端的可靠性投递
- 保障消息的成功发出
- 保障MQ节点的成功接收
- 发送端收到MQ节点（Broker）确认应答
- 完善的消息进行补偿机制
大厂的解决方案一
- 消息落库，对消息状态进行打标
各步骤的解释： - 第一步：把业务（比如写订单）入库，消息也入库，主要记录消息是什么和状态
- 第二步：发送消息
- 第三步：若Broker收到消息，返回确认
- 第四步：更新消息状态
- 第五步：分布式的定时任务，如果监听到某时间内消息状态还是没收到确认的状态就尝试重试操作
- 第六步：重试操作
- 第七步：如果重试次数大于3次，表明存在别的问题了（比如配置问题），设置状态值为2，如有必要增加补偿系统进行处理
大厂的解决方案二
- 保障MQ可靠性投递的第一种解决方案在高并发场景下并不合适，因为进行了消息和业务信息的入库操作，数据库有瓶颈
- 消息的延迟投递、做二次确认。回调检查
各步骤的解释：
- Upstream：上游服务，生产者
- DownStream：下游服务，消费者
- 第一步：先业务数据入库，然后第一次发送消息
- 第二步：延迟再发送这个消息（延迟投递）、注意和第一次发的消息不放在一个队列里面
- 第三步：发送给消费者处理
- 第四步：消费者若收到生产者发来的消息也生成一条消息发送到MQ（并不是直接响应ack）
- 第五步：CallbackService监听消费者发来的消息，把这个消息做持久化处理
- 第六步：CallbackService监听生产者发来的延迟消息，检查消息数据库看第一次发送的消息有没有被处理好，如果CallBackService发现消息还被写入数据库，向生产者发送重新发送的命令
3.2幂等性和避免重复消费
- 幂等性简单来说就是多次执行，结果不变
- 消费端实现幂等性，就意味着，我们的消息永远不会消费多次，即使我们收到了多条一样的消息
主流的幂等性操作一
- 唯一性ID+指纹码机制，利用数据库主键去重
- select count(1) from order where id = 唯一性ID+指纹码（一些规则）
- 好处：简单
- 坏处：高并发下有数据库写入的性能瓶颈
- 解决方案：根据ID进行分库分表进行路由算法
主流的幂等性操作二
- 利用Redis的原子性去实现
3.3Confirm确认消息
理解Confirm消息确认机制
- 消息的确认，是指生产者投递消息后，如果Broker收到消息，则会给我们生产者一个应答
- 生产者进行接收应答，用来确定这条消息是否正常的发送到Broker，这种方式也是消息的可靠性投递的保障
Confirm的实现
- 第一步：在channel上开启确认模式，channel.confirmSelect();
- 第二步：在channel上添加监听，addConfirmListener,监听成功和失败的返回结果，根据句具体的结果对消息进行重新发送，或记录日志等后续操作
代码示例
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
//1 创建ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
//2 获取C onnection
Connection connection = connectionFactory.newConnection();
//3 通过Connection创建一个新的Channel
Channel channel = connection.createChannel();
//4 指定我们的消息投递模式: 消息的确认模式
channel.confirmSelect();
String exchangeName = "test_confirm_exchange";
String routingKey = "confirm.save";
//5 发送一条消息
String msg = "Hello RabbitMQ Send confirm message!";
channel.basicPublish(exchangeName, routingKey, null, msg.getBytes());
//6 添加一个确认监听
channel.addConfirmListener(new ConfirmListener() {
//deliveryTag就是消息的唯一标签
@Override
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
System.err.println("-------no ack!-----------");
}
@Override
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
System.err.println("-------ack!-----------");
}
});
}
}
消费者代码
public class Consumer {
public static void main(String[] args) throws Exception {
//1 创建ConnectionFactory
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
//2 获取C onnection
Connection connection = connectionFactory.newConnection();
//3 通过Connection创建一个新的Channel
Channel channel = connection.createChannel();
String exchangeName = "test_confirm_exchange";
String routingKey = "confirm.#";
String queueName = "test_confirm_queue";
//4 声明交换机和队列 然后进行绑定设置, 最后制定路由Key
channel.exchangeDeclare(exchangeName, "topic", true);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchangeName, routingKey);
//5 创建消费者
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
channel.basicConsume(queueName, true, queueingConsumer);
while(true){
Delivery delivery = queueingConsumer.nextDelivery();
String msg = new String(delivery.getBody());
System.err.println("消费端: " + msg);
}
}
}
3.4Return消息机制
- Return Listener用于处理一些不可路由的消息
- 我们的消息生产者，通过指定一个Exchange和RoutingKey，把消息送达到某一个队列当中，然后我们的消费者监听队列，进行消费处理操作
- 但是在某些情况下，如果我们在发送消息的时候，当前的exchange不存在或者指定的路由key路由不到，这个时候如果我们需要监听这种不可达的消息，就要使用Return Listner
- 在基础API中有个关键配置项–Mandatory，如果为true监听器会接收到路由不可达的消息，然后进行后续处理，如果为false，那么broker端会自动删除该消息
代码演示
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchange = "test_return_exchange";
String routingKey = "return.save";
String routingKeyError = "abc.save";
String msg = "Hello RabbitMQ Return Message";
channel.addReturnListener(new ReturnListener() {
@Override
public void handleReturn(int replyCode, String replyText, String exchange,
String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.err.println("---------handle return----------");
System.err.println("replyCode: " + replyCode);
System.err.println("replyText: " + replyText);
System.err.println("exchange: " + exchange);
System.err.println("routingKey: " + routingKey);
System.err.println("properties: " + properties);
System.err.println("body: " + new String(body));
}
});
channel.basicPublish(exchange, routingKeyError, true, null, msg.getBytes());
//channel.basicPublish(exchange, routingKeyError, true, null, msg.getBytes());
}
}
消费者代码
public class Consumer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "test_return_exchange";
String routingKey = "return.#";
String queueName = "test_return_queue";
channel.exchangeDeclare(exchangeName, "topic", true, false, null);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchangeName, routingKey);
QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
channel.basicConsume(queueName, true, queueingConsumer);
while(true){
Delivery delivery = queueingConsumer.nextDelivery();
String msg = new String(delivery.getBody());
System.err.println("消费者: " + msg);
}
}
}
3.5Confirm消费端自定义监听
- 我们一般在代码中编写while循环，进行consumer.nextDelivery方法获取下一条消息，然后进行消费处理
- 使用自定义Consumer更加方便，解耦性更强，实际工作中使用
代码演示
- 主要看消费者
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchange = "test_consumer_exchange";
String routingKey = "consumer.save";
String msg = "Hello RabbitMQ Consumer Message";
for(int i =0; i<5; i ++){
channel.basicPublish(exchange, routingKey, true, null, msg.getBytes());
}
}
}
MyConsumer
public class MyConsumer extends DefaultConsumer {
public MyConsumer(Channel channel) {
super(channel);
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.err.println("-----------consume message----------");
System.err.println("consumerTag: " + consumerTag);
System.err.println("envelope: " + envelope);
System.err.println("properties: " + properties);
System.err.println("body: " + new String(body));
}
}
Consumer
public class Consumer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "test_consumer_exchange";
String routingKey = "consumer.#";
String queueName = "test_consumer_queue";
channel.exchangeDeclare(exchangeName, "topic", true, false, null);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchangeName, routingKey);
channel.basicConsume(queueName, true, new MyConsumer(channel));
}
}
3.6消费端限流
- 假设一个场景，首先我们RabbitMQ服务器有成千上万条未处理的消息，我们随便打开一个消费者客户端，后果很严重
- 巨量信息涌入，单个消费者无法处理
- RabbitMQ提供一种qos（服务质量保证）功能，即在非自动确认消息的前提下，如果一定数目的消息（通过基于consume或者channel设置Qos的值）未被确认前，不进行消费新的消息
- void BasicQos(uint prefetchSize,ushort prefetchCount,bool global) //global：true-channel级别进行限制，false-消费者级别限制
- prefetchSize：0
- prefetchCount：会告诉RabbitMQ不要同时给一个消费者推送多于N个消息，即一旦有N个消息还没有ack，则该consumer将block掉，知道消息ack
- global：true/false是否将上面设置应用于channel简单点说，就是上面限制是channel级别还是consumer级别
- prefetchSize和global这两项，rabbitMQ没有实现，暂且不研究prefetch_count在no_ask=false的情况下生效，即在自动应答的情况下这两个值不生效
代码演示
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchange = "test_qos_exchange";
String routingKey = "qos.save";
String msg = "Hello RabbitMQ QOS Message";
for(int i =0; i<5; i ++){
channel.basicPublish(exchange, routingKey, true, null, msg.getBytes());
}
}
}
消费者代码
MyConsumer：注意消费后进行确认操作
public class MyConsumer extends DefaultConsumer {
private Channel channel ;
public MyConsumer(Channel channel) {
super(channel);
this.channel = channel;
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.err.println("-----------consume message----------");
System.err.println("consumerTag: " + consumerTag);
System.err.println("envelope: " + envelope);
System.err.println("properties: " + properties);
System.err.println("body: " + new String(body));
//进行确认操作
channel.basicAck(envelope.getDeliveryTag(), false);
}
}
Consumer
public class Consumer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "test_qos_exchange";
String queueName = "test_qos_queue";
String routingKey = "qos.#";
channel.exchangeDeclare(exchangeName, "topic", true, false, null);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchangeName, routingKey);
//1 限流方式 第一件事就是 autoAck设置为 false: queueName, false, new MyConsumer(channel)
channel.basicQos(0, 1, false);
channel.basicConsume(queueName, false, new MyConsumer(channel));
}
}
3.7消费端ACK与重回队列
String basicConsume(String queue, boolean autoAck, Consumer callback)
消费端的手工ACK和NACK
- 消费端进行消费的时候，如果由于业务异常我们可以进行日志的记录，然后进行补偿
- 如果出现消费者服务器宕机等严重问题，就需要手工ACK保障消费端消费成功
消费端重回队列
- 消费端重回队列为了对没有处理成功的消息，把消息重新会递给Broker
- 但是一般都会关闭这个功能
代码演示
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchange = "test_ack_exchange";
String routingKey = "ack.save";
for(int i =0; i<5; i ++){
Map<String, Object> headers = new HashMap<String, Object>();
headers.put("num", i);
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.deliveryMode(2)
.contentEncoding("UTF-8")
.headers(headers)
.build();
String msg = "Hello RabbitMQ ACK Message " + i;
channel.basicPublish(exchange, routingKey, true, properties, msg.getBytes());
}
}
}
消费者代码
public class MyConsumer extends DefaultConsumer {
private Channel channel ;
public MyConsumer(Channel channel) {
super(channel);
this.channel = channel;
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.err.println("-----------consume message----------");
System.err.println("body: " + new String(body));
try {
Thread.sleep(2000);
} catch (InterruptedException e) {
e.printStackTrace();
}
if((Integer)properties.getHeaders().get("num") == 0) {
//long deliveryTag, boolean multiple, boolean requeue
channel.basicNack(envelope.getDeliveryTag(), false, true);
} else {
channel.basicAck(envelope.getDeliveryTag(), false);
}
}
}
public class Consumer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchangeName = "test_ack_exchange";
String queueName = "test_ack_queue";
String routingKey = "ack.#";
channel.exchangeDeclare(exchangeName, "topic", true, false, null);
channel.queueDeclare(queueName, true, false, false, null);
channel.queueBind(queueName, exchangeName, routingKey);
// 手工签收 必须要关闭 autoAck = false
channel.basicConsume(queueName, false, new MyConsumer(channel));
}
}
3.8TTL队列/消息
TTL
- TTL是 time to live的缩写，也就是生存时间
- RabbitMQ支持消息的过期时间，在消息发送时可以指定
- RabbitMQ支持队列的过期时间，从消息入队列开始计算，只要超过了队列的超时时间配置，那么消息就会自动清除
- 有两种设置方式一是创建新队列时指定TTL参数如下图，另一个是发送消息时在附带的参数中增加过期时间。
3.9死信队列
- 死信队列，DLX，Dead-Letter-Exchange
- 利用DLX，当一条消息在一个队列中变成死信（dead message）之后，它能重新publish到另一个Exchange，这个Exchange就是DLX。
- DLX也是一个正常的Exchange，和一般Exchange没有区别，它能在任何的队列上被指定，实际上就是设置某个队列的属性
- 当这个队列中有死信时，RabbitMQ就会自动的将这个消息重新发布到设置的Exchange上去，进而被路由到另一个队列
- 可以监听这个队列中消息做相应的处理，这个特性可以弥补RabbitMQ3.0以前支持的immediate参数功能
消息变为死信的几种情况
- 消息被拒绝（basic.reject/basic.nack）并且requeue=false（不重回队列）
- 消息TTL过期
- 队列达到最大长度，又有消息过来
死信队列的设置
- 首先要设置死信队列的Exchange和Queue，然后进行绑定
- Exchange：dlx.exchange
- Queue：dlx.queue
- RoutingKey：#
- 然后进行正常声明交换机、队列、绑定、只不过我们需要在队列上加一个参数
- arguments.put(“x-dead-letter-exchange”,“dlx.exchange”)
代码演示
生产者代码
public class Producer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
String exchange = "test_dlx_exchange";
String routingKey = "dlx.save";
String msg = "Hello RabbitMQ DLX Message";
for(int i =0; i<1; i ++){
AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
.deliveryMode(2)
.contentEncoding("UTF-8")
.expiration("10000")
.build();
channel.basicPublish(exchange, routingKey, true, properties, msg.getBytes());
}
}
}
消费者代码
public class MyConsumer extends DefaultConsumer {
public MyConsumer(Channel channel) {
super(channel);
}
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.err.println("-----------consume message----------");
System.err.println("consumerTag: " + consumerTag);
System.err.println("envelope: " + envelope);
System.err.println("properties: " + properties);
System.err.println("body: " + new String(body));
}
}
public class Consumer {
public static void main(String[] args) throws Exception {
ConnectionFactory connectionFactory = new ConnectionFactory();
connectionFactory.setHost("192.168.11.76");
connectionFactory.setPort(5672);
connectionFactory.setVirtualHost("/");
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
// 这就是一个普通的交换机 和 队列 以及路由
String exchangeName = "test_dlx_exchange";
String routingKey = "dlx.#";
String queueName = "test_dlx_queue";
channel.exchangeDeclare(exchangeName, "topic", true, false, null);
Map<String, Object> agruments = new HashMap<String, Object>();
agruments.put("x-dead-letter-exchange", "dlx.exchange");
//这个agruments属性，要设置到声明队列上
channel.queueDeclare(queueName, true, false, false, agruments);
channel.queueBind(queueName, exchangeName, routingKey);
//要进行死信队列的声明:
channel.exchangeDeclare("dlx.exchange", "topic", true, false, null);
channel.queueDeclare("dlx.queue", true, false, false, null);
channel.queueBind("dlx.queue", "dlx.exchange", "#");
channel.basicConsume(queueName, true, new MyConsumer(channel));
}
}
这个代码如果不进行消费者消费那么由于生产者消息ttl设置为1000过期后会到死信队列上去
扫一扫
3275
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
