RabbitMQ:消息丢失处理

最新推荐文章于 2024-05-18 16:29:21 发布

小纸人呀

最新推荐文章于 2024-05-18 16:29:21 发布

阅读量633

点赞数

分类专栏： RabbitMQ 文章标签： rabbitmq

本文链接：https://blog.csdn.net/xzr1117/article/details/115454580

版权

RabbitMQ 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

1. 本篇概要

RabbitMQ存在数据丢失的三种情况：

①生产者发送消息失败：写消息的过程中，消费还没到MQ，在网络传输过程中丢了，或者是消费到了RabbitMQ，但是MQ那出错了，没保存下来。

② 中间件挂了：RabbitMQ接收到消费之后，先暂存在自己的内存里，结果消费者还没来得及消费，RabbitMQ自己挂掉了，导致暂存在内存中的数据丢了。

③ 消费者弄丢数据：消费了拿到了这个数据，但还没来得及处理，自己就挂掉了。但是RabbitMQ以为这个消费者消费完了。

2. 生产者发送消息失败

2.1 RabbitMQ的事务

/**
	 * 发送消息到RabbitMQ,执行数据库减库存和订单生成
	 * @param username
	 * @param productNo
	 * @param spikeQuantity
	 */
	private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
		ConnectionFactory factory = new ConnectionFactory();  
		factory.setHost("127.0.0.1");
		factory.setUsername("guest");
		factory.setPassword("guest");
		factory.setPort(5672);
		Channel channel = null;
		try {
			Connection connection = factory.newConnection();
			channel = connection.createChannel();
			BlockingQueue<String> blockingQueue = new ArrayBlockingQueue<String>(100);	
			//开启事务
			channel.txSelect();
			//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
			channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
			//创建一个type=direct 持久化的 非自动删除的交换器
			channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
			//将交换器与队列通过路由键绑定
			channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
			//生成全局唯一的messageID
			String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
			//设置消息的请求头部参数
	        BasicProperties bp = new BasicProperties();
	        bp.builder().deliveryMode(2).messageId(messageid);
	        //设置消息体
	        Map<String, String> map = new HashMap<>();
	        map.put("username", username);
	        map.put("productNo", productNo);
	        map.put("spikeQuantity", spikeQuantity);
	        	
	        //发送一条持久化的消息
			channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());		 
            //提交事务			
            channel.txCommit();	
			//关闭连接
	        channel.close();
	        connection.close();
	        return "发送成功";
			
		} catch (Exception e) {
	        //回滚事务
			//channel.txRollback();
			return "消息发送失败";
		}
		
	}

缺点：RabbitMQ事务太耗性能，基本上吞吐量会下来。事务机制是同步的，生产者发送这个消息，会同步阻塞卡住，等待是成功还是失败，会导致生产者发送消息的吞吐量下降。

2.2 生产者的确认机制confirm(同步)

/**
	 * 发送消息到RabbitMQ,执行数据库减库存和订单生成
	 * @param username
	 * @param productNo
	 * @param spikeQuantity
	 */
	private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
		ConnectionFactory factory = new ConnectionFactory();  
		factory.setHost("127.0.0.1");
		factory.setUsername("guest");
		factory.setPassword("guest");
		factory.setPort(5672);
		Channel channel = null;
		try {
			Connection connection = factory.newConnection();
			channel = connection.createChannel();
			BlockingQueue<String> blockingQueue = new ArrayBlockingQueue<String>(100);
			//将信道设置成confirm模式
			channel.confirmSelect();
			//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
			channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
			//创建一个type=direct 持久化的 非自动删除的交换器
			channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
			//将交换器与队列通过路由键绑定
			channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
			//生成全局唯一的messageID
			String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
			//设置消息的请求头部参数
	        BasicProperties bp = new BasicProperties();
	        bp.builder().deliveryMode(2).messageId(messageid);
	        //设置消息体
	        Map<String, String> map = new HashMap<>();
	        map.put("username", username);
	        map.put("productNo", productNo);
	        map.put("spikeQuantity", spikeQuantity);
	        //发送一条持久化的消息
			channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());
			// 将发送出去的消息存入缓存中，缓存可以是一个ArrayList或者BlockingQueue之类的
			blockingQueue.add(map.toString());
				
			//消息发送方确认
				
			if (channel.waitForConfirms()) {
	            System.out.println("send message success");
	            // 将缓存中的消息清空
                blockingQueue.clear();
	        } else {
	            System.out.println("send message failed");
	            // 将缓存中的消息重新发送
	        }
			//关闭连接
	        channel.close();
	        connection.close();
	        return "发送成功";
			
		} catch (Exception e) {
			return "消息发送失败";
		}
		
	}

事务机制和publisher confirm机制是互斥的，不能共存

2.3 异步confirm

/**
	 * 发送消息到RabbitMQ,执行数据库减库存和订单生成
	 * @param username
	 * @param productNo
	 * @param spikeQuantity
	 */
	private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
		ConnectionFactory factory = new ConnectionFactory();  
		factory.setHost("127.0.0.1");
		factory.setUsername("guest");
		factory.setPassword("guest");
		factory.setPort(5672);
		Channel channel = null;
		try {
			Connection connection = factory.newConnection();
			channel = connection.createChannel();
			//将信道设置成confirm模式
			channel.confirmSelect();
			SortedSet<Long> confirmSet = new TreeSet<Long>();
			channel.addConfirmListener(new ConfirmListener() {
	            @Override
	            public void handleAck(long deliveryTag, boolean multiple) throws IOException {
	                System.out.println("Ack,SeqNo：" + deliveryTag + ",multiple：" + multiple);
	                if (multiple) {
	                    confirmSet.headSet(deliveryTag - 1).clear();
	                } else {
	                    confirmSet.remove(deliveryTag);
	                }
	            }

	            @Override
	            public void handleNack(long deliveryTag, boolean multiple) throws IOException {
	                System.out.println("Nack,SeqNo：" + deliveryTag + ",multiple：" + multiple);
	                if (multiple) {
	                    confirmSet.headSet(deliveryTag - 1).clear();
	                } else {
	                    confirmSet.remove(deliveryTag);
	                }
	                // 注意这里需要添加处理消息重发的场景
	            }
	        });
			
			//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
			channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
			//创建一个type=direct 持久化的 非自动删除的交换器
			channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
			//将交换器与队列通过路由键绑定
			channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
			//生成全局唯一的messageID
			String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
			//设置消息的请求头部参数
	        BasicProperties bp = new BasicProperties();
	        bp.builder().deliveryMode(2).messageId(messageid);
	        //设置消息体
	        Map<String, String> map = new HashMap<>();
	        map.put("username", username);
	        map.put("productNo", productNo);
	        map.put("spikeQuantity", spikeQuantity);
	        
	        long nextSeqNo = channel.getNextPublishSeqNo();
        	//发送一条持久化的消息
			channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());
			confirmSet.add(nextSeqNo);
			//关闭连接
	        channel.close();
	        connection.close();
	        return "发送成功";
			
		} catch (Exception e) {
			return "消息发送失败";
		}
		
	}

2.4 SpringBoot整合RabbitMQ的rabbitTemplate的写法

发送消息

/**
	 * 直连交换机
	 * @param username
	 * @param orderNo
	 */
	public String ToReduceInventoryOfDirectExchange(String username, String orderNo) {
		String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
        MessageOfOrder msg = new MessageOfOrder();
        msg.setId(messageid);
        msg.setUserName(username);
        msg.setOrderNo(orderNo);
        msg.setFailureCount(0);
        String jmsg=JSON.toJSONString(msg);
        //q使用自定义的数据对象
        CorrelationDataExt correlationData = new CorrelationDataExt();
        correlationData.setId(messageid);
        correlationData.setData(jmsg);
        correlationData.setQueueName(CommonConstant.OrderToInventoryQueue);
        correlationData.setExchangeName(CommonConstant.OrderToInventoryExchage);
        correlationData.setRoutName(CommonConstant.OrderToInventoryRout);
        rabbitTemplate.setEncoding("UTF-8");
        //设置消息的请求头部参数，匿名内部类
        
        MessagePostProcessor mpp = new MessagePostProcessor() {
			@Override
			public Message postProcessMessage(Message message) throws AmqpException {
				 message.getMessageProperties().setMessageId(messageid);
				 //设置消息持久化
				 message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
				 return message;
			}
        };
        try {
        	rabbitTemplate.convertAndSend(CommonConstant.OrderToInventoryExchage, CommonConstant.OrderToInventoryRout, jmsg,mpp);
        	
		}catch(Exception e) {
			//q保存报错信息到DB
			String key = CommonConstant.ReduceInventoryFailure;
			
			jedisCluster.rpush(key, jmsg);
			return CommonConstant.failure;
		}
        return messageid;

	}

消息监听

@Bean
	public RabbitTemplate createRabbitTemplate(ConnectionFactory connectionFactory) {
		RabbitTemplate rabbitTemplate = new RabbitTemplate();

		
		rabbitTemplate.setConnectionFactory(connectionFactory);
		// q设置开启Mandatory,才能触发回调函数,无论消息推送结果怎么样都强制调用回调函数
		rabbitTemplate.setMandatory(true);

		// 消息推送到server，但是在server里找不到交换机
		rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
			@Override
			public void confirm(CorrelationData correlationData, boolean ack, String cause) {
				CorrelationDataExt correlationDataExt = (CorrelationDataExt) correlationData;
				if (!ack) {
					if (correlationData instanceof CorrelationDataExt) {
						sendToRedis(correlationDataExt.getId(),correlationDataExt.getExchangeName(),correlationDataExt.getRoutName(),correlationDataExt.getData());
					}
				}else {
					//判断之前是否有发送失败的，有的话删除
					checkExistRedisWhenSendSuccess(correlationDataExt.getId());
				}

			}
		});

		// 消息推送到server，找到交换机了，但是没找到队列
		rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
			@Override
			public void returnedMessage(Message message, int replyCode, String replyText, String exchange,
					String routingKey) {
				
				sendToRedis(message.getMessageProperties().getMessageId(),exchange,routingKey,new String(message.getBody()));
				
			}
		});

		return rabbitTemplate;
	}


/**
	 * 写入Redis做定时插入
	 * @param messageId
	 * @param exchange
	 * @param routkey
	 * @param body
	 */
	public void sendToRedis(String messageId,String exchange,String routkey,Object body) {
		SendFailureMsg sfm = new SendFailureMsg();
		redisConfig.getJedisCluster().getClusterNodes();
		JedisCluster redis = redisConfig.getJedisCluster();

		if (redis.exists(failureKey)) {
			String FailureMessageJson = jedisCluster.get(failureKey);
			ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
			messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
					SendFailureMsg.class);
			// q重新入队
			if (messageOfOrderList.size() > 0) {
				for (SendFailureMsg moo : messageOfOrderList) {
					if (moo.getMessageId().equals(messageId)) {
						String jvalue = CommonUtils.ObjectToJson(moo);
						jedisCluster.lrem(failureKey, 1, jvalue);
						moo.setFailureCount(moo.getFailureCount() + 1);
						jedisCluster.rpush(failureKey, JSON.toJSONString(moo));
						existInRedis = true;
						break;
					}
				}
			}
		}

		if (!existInRedis) {
			sfm.setBody(body);
			sfm.setExchange(exchange);
			sfm.setRout(routkey);
			sfm.setMessageId(messageId);
			sfm.setFailureCount(0);

			// q保存报错信息到DB
			String key = CommonConstant.ReduceInventoryFailure;

			jedisCluster.rpush(key, JSON.toJSONString(sfm));
		}
	}
	
	/**
	 * 发送成功，则检查是否有失败过，如果有就将其删除
	 * @param messageId
	 */
	public void checkExistRedisWhenSendSuccess(String messageId) {
		
		redisConfig.getJedisCluster().getClusterNodes();
		JedisCluster redis = redisConfig.getJedisCluster();
		if (redis.exists(failureKey)) {
			String FailureMessageJson = jedisCluster.get(failureKey);
			ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
			messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
					SendFailureMsg.class);
			// q重新入队
			if (messageOfOrderList.size() > 0) {
				for (SendFailureMsg moo : messageOfOrderList) {
					if (moo.getMessageId().equals(messageId)) {
						String jvalue = CommonUtils.ObjectToJson(moo);
						jedisCluster.lrem(failureKey, 1, jvalue);
						break;
					}
				}
			}
		}

	}

写个定时任务自动处理发送失败的消息

@Component
@EnableScheduling
@EnableAsync
public class MultithreadScheduleTask {

	@Autowired
	RabbitTemplate rabbitTemplate; // q使用RabbitTemplate,这提供了接收/发送等等方法

	@Autowired
	private JedisCluster jedisCluster;
	@Autowired
	private RedisConfig redisConfig;


	@Async
	@Scheduled(fixedDelay = 1000 * 60 * 30)
	public void ReduceInventoryFailureMessage() throws InterruptedException {
		System.out.println(
				"第一个定时任务开始 : " + LocalDateTime.now().toLocalTime() + "\r\n线程 : " + Thread.currentThread().getName());

		redisConfig.getJedisCluster().getClusterNodes();
		JedisCluster redis = redisConfig.getJedisCluster();
		if (redis.exists(CommonConstant.ReduceInventoryFailure)) {

			String FailureMessageJson = jedisCluster.get(CommonConstant.ReduceInventoryFailure);
			ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
			messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
					SendFailureMsg.class);
			// q重新入队
			if (messageOfOrderList.size() > 0) {
				for (SendFailureMsg moo : messageOfOrderList) {
					try {

						if (Integer.valueOf(moo.getFailureCount()) > 5) {
							ResendDel(CommonConstant.ReduceInventoryFailure,moo);
							
						} else {
							MessagePostProcessor mpp = new MessagePostProcessor() {
								@Override
								public Message postProcessMessage(Message message) throws AmqpException {
									message.getMessageProperties().setMessageId(moo.getMessageId());
									return message;
								}
							};
							rabbitTemplate.convertAndSend(moo.getExchange(), moo.getRout(), moo.getBody(), mpp);

						}
					} catch (Exception e) {
						e.printStackTrace();
					}

				}
			}
		}

	}
	

	/**
	 * 重新发送成功或失败次数超过5次则从缓存删除数据
	 * 
	 * @param id
	 * @param moo
	 * @param reduceinventoryfailure
	 */
	public void ResendDel(String key, SendFailureMsg value) {
		String jvalue = CommonUtils.ObjectToJson(value);
		jedisCluster.lrem(key, 1, jvalue);

	}

	/**
	 * 重写发送
	 * 
	 * @param id
	 */
	public void ResendFailed(String key, SendFailureMsg moo) {
		MessagePostProcessor mpp = new MessagePostProcessor() {
			@Override
			public Message postProcessMessage(Message message) throws AmqpException {
				message.getMessageProperties().setMessageId(moo.getMessageId());
				return message;
			}
		};
		rabbitTemplate.convertAndSend(moo.getExchange(), moo.getRout(), moo.getBody(), mpp);
	}

3.中间件挂了弄丢了消息

开启RabbitMQ队列持久化和消息持久化

3.1队列持久化

Bean创建队列

@Bean
public Queue DirectQueue() {
//第二个参数为true,队列持久化
	return new Queue(CommonConstant.OrderToInventoryQueue, true);
}

channel创建队列

Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
//第二个参数：是否持久化；排他队列；自动删除
channel.queueDeclare(CommonConstant.OrderToInventoryQueue, true, false, false, null);

3.2消息持久化

设置消息头部信息deliverymode=2，deliverymode=1是消息不持久化

3.2.1springboot整合RabbitMQ的rabbitTemplate

MessageDeliveryMode.PERSISTENT 返回2

MessageDeliveryMode.NON_PERSISTENT 返回1

 //设置消息的请求头部参数，匿名内部类
        
 MessagePostProcessor mpp = new MessagePostProcessor() {
	@Override
	public Message postProcessMessage(Message message) throws AmqpException {
		//设置消息持久化
		 message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
				 return message;
			}
        };
        
     rabbitTemplate.convertAndSend(CommonConstant.OrderToInventoryExchage, CommonConstant.OrderToInventoryRout, jmsg,mpp);

3.2.2channel发布持久化消息

MessageProperties.PERSISTENT_TEXT_PLAIN 返回2

MessageProperties.TEXT_PLAIN 返回1

//设置第三个参数deliverymode为2
channel.basicPublish("exchange", "routKey", MessageProperties.PERSISTENT_TEXT_PLAIN, messagebody.getBytes());

4.消费者弄丢了消息

消费者手动确认机制

container.setAcknowledgeMode(AcknowledgeMode.MANUAL); // RabbitMQ默认是自动确认，这里改为手动确认消息
Boolean excute = true;
try {
//消费者消费过程
	result = inventoryReduceApi.reduceInventory(message, messageId);

} catch (Exception e) {
	excute = false;
} finally {
	if (excute) {
        //手动确认
		channel.basicAck(deliveryTag, true);

	} else {
		// 第二个参数是否requeue，true则重新入队列，否则丢弃或者进入死信队列
		channel.basicReject(deliveryTag, false);
		// 将消息放到队尾
		channel.basicPublish(exchange, rout, MessageProperties.PERSISTENT_TEXT_PLAIN, message.getBytes());
	}
				
}

小纸人呀

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
RabbitMQ:消息丢失处理

RabbitMQ存在数据丢失的三种情况： ①生产者发送消息失败：写消息的过程中，消费还没到MQ，在网络传输过程中丢了，或者是消费到了RabbitMQ，但是MQ那出错了，没保存下来。②中间件挂了：RabbitMQ接收到消费之后，先暂存在自己的内存里，结果消费者还没来得及消费，RabbitMQ自己挂掉了，导致暂存在内存中的数据丢了。③消费者弄丢数据：消费了拿到了这个数据，但还没来得及处理，自己就挂掉了。但是RabbitMQ以为这个消费者消费完了。...
复制链接

扫一扫