本文详细介绍了RabbitMQ中防止数据丢失的三种情况：生产者发送失败、中间件挂掉和消费者弄丢数据。针对这些情况，文章讨论了RabbitMQ的事务、确认机制（同步和异步）以及SpringBoot中rabbitTemplate的使用方法。同时，提到了队列和消息的持久化配置，以及消费者的手动确认机制，确保消息的可靠传递。
1. 本篇概要
RabbitMQ存在数据丢失的三种情况：
①生产者发送消息失败： 写消息的过程中，消费还没到MQ，在网络传输过程中丢了，或者是消费到了RabbitMQ，但是MQ那出错了，没保存下来。
② 中间件挂了：RabbitMQ接收到消费之后，先暂存在自己的内存里，结果消费者还没来得及消费，RabbitMQ自己挂掉了，导致暂存在内存中的数据丢了。
③ 消费者弄丢数据：消费了拿到了这个数据，但还没来得及处理，自己就挂掉了。但是RabbitMQ以为这个消费者消费完了。
2. 生产者发送消息失败
2.1 RabbitMQ的事务
/**
* 发送消息到RabbitMQ,执行数据库减库存和订单生成
* @param username
* @param productNo
* @param spikeQuantity
*/
private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setUsername("guest");
factory.setPassword("guest");
factory.setPort(5672);
Channel channel = null;
try {
Connection connection = factory.newConnection();
channel = connection.createChannel();
BlockingQueue<String> blockingQueue = new ArrayBlockingQueue<String>(100);
//开启事务
channel.txSelect();
//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
//创建一个type=direct 持久化的 非自动删除的交换器
channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
//将交换器与队列通过路由键绑定
channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
//生成全局唯一的messageID
String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
//设置消息的请求头部参数
BasicProperties bp = new BasicProperties();
bp.builder().deliveryMode(2).messageId(messageid);
//设置消息体
Map<String, String> map = new HashMap<>();
map.put("username", username);
map.put("productNo", productNo);
map.put("spikeQuantity", spikeQuantity);
//发送一条持久化的消息
channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());
//提交事务
channel.txCommit();
//关闭连接
channel.close();
connection.close();
return "发送成功";
} catch (Exception e) {
//回滚事务
//channel.txRollback();
return "消息发送失败";
}
}
缺点：RabbitMQ事务太耗性能，基本上吞吐量会下来。事务机制是同步的，生产者发送这个消息，会同步阻塞卡住，等待是成功还是失败，会导致生产者发送消息的吞吐量下降。
2.2 生产者的确认机制confirm(同步)
/**
* 发送消息到RabbitMQ,执行数据库减库存和订单生成
* @param username
* @param productNo
* @param spikeQuantity
*/
private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setUsername("guest");
factory.setPassword("guest");
factory.setPort(5672);
Channel channel = null;
try {
Connection connection = factory.newConnection();
channel = connection.createChannel();
BlockingQueue<String> blockingQueue = new ArrayBlockingQueue<String>(100);
//将信道设置成confirm模式
channel.confirmSelect();
//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
//创建一个type=direct 持久化的 非自动删除的交换器
channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
//将交换器与队列通过路由键绑定
channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
//生成全局唯一的messageID
String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
//设置消息的请求头部参数
BasicProperties bp = new BasicProperties();
bp.builder().deliveryMode(2).messageId(messageid);
//设置消息体
Map<String, String> map = new HashMap<>();
map.put("username", username);
map.put("productNo", productNo);
map.put("spikeQuantity", spikeQuantity);
//发送一条持久化的消息
channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());
// 将发送出去的消息存入缓存中，缓存可以是一个ArrayList或者BlockingQueue之类的
blockingQueue.add(map.toString());
//消息发送方确认
if (channel.waitForConfirms()) {
System.out.println("send message success");
// 将缓存中的消息清空
blockingQueue.clear();
} else {
System.out.println("send message failed");
// 将缓存中的消息重新发送
}
//关闭连接
channel.close();
connection.close();
return "发送成功";
} catch (Exception e) {
return "消息发送失败";
}
}事务机制和publisher confirm机制是互斥的，不能共存
2.3 异步confirm
/**
* 发送消息到RabbitMQ,执行数据库减库存和订单生成
* @param username
* @param productNo
* @param spikeQuantity
*/
private String SendRabbitMQMessageOfSpike(String username, String productNo, String spikeQuantity) {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("127.0.0.1");
factory.setUsername("guest");
factory.setPassword("guest");
factory.setPort(5672);
Channel channel = null;
try {
Connection connection = factory.newConnection();
channel = connection.createChannel();
//将信道设置成confirm模式
channel.confirmSelect();
SortedSet<Long> confirmSet = new TreeSet<Long>();
channel.addConfirmListener(new ConfirmListener() {
@Override
public void handleAck(long deliveryTag, boolean multiple) throws IOException {
System.out.println("Ack,SeqNo：" + deliveryTag + ",multiple：" + multiple);
if (multiple) {
confirmSet.headSet(deliveryTag - 1).clear();
} else {
confirmSet.remove(deliveryTag);
}
}
@Override
public void handleNack(long deliveryTag, boolean multiple) throws IOException {
System.out.println("Nack,SeqNo：" + deliveryTag + ",multiple：" + multiple);
if (multiple) {
confirmSet.headSet(deliveryTag - 1).clear();
} else {
confirmSet.remove(deliveryTag);
}
// 注意这里需要添加处理消息重发的场景
}
});
//队列名称；是否持久化；是否消费者消费完自动删除；是否排外，true为一个队列只有一个消费者
channel.queueDeclare(CommonConstant.SpikeProductQueue, true, false, false, null);
//创建一个type=direct 持久化的 非自动删除的交换器
channel.exchangeDeclare(CommonConstant.SpikeProductExchage, CommonConstant.direct ,true, false, null);
//将交换器与队列通过路由键绑定
channel.queueBind(CommonConstant.SpikeProductQueue, CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout);
//生成全局唯一的messageID
String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
//设置消息的请求头部参数
BasicProperties bp = new BasicProperties();
bp.builder().deliveryMode(2).messageId(messageid);
//设置消息体
Map<String, String> map = new HashMap<>();
map.put("username", username);
map.put("productNo", productNo);
map.put("spikeQuantity", spikeQuantity);
long nextSeqNo = channel.getNextPublishSeqNo();
//发送一条持久化的消息
channel.basicPublish(CommonConstant.SpikeProductExchage, CommonConstant.SpikeProductRout, bp , map.toString().getBytes());
confirmSet.add(nextSeqNo);
//关闭连接
channel.close();
connection.close();
return "发送成功";
} catch (Exception e) {
return "消息发送失败";
}
}2.4 SpringBoot整合RabbitMQ的rabbitTemplate的写法
发送消息
/**
* 直连交换机
* @param username
* @param orderNo
*/
public String ToReduceInventoryOfDirectExchange(String username, String orderNo) {
String messageid = String.valueOf(UUID.randomUUID()).replace("-", "").trim()+CommonUtils.getSystemDate(CommonConstant.DateFormatOfymdhms);
MessageOfOrder msg = new MessageOfOrder();
msg.setId(messageid);
msg.setUserName(username);
msg.setOrderNo(orderNo);
msg.setFailureCount(0);
String jmsg=JSON.toJSONString(msg);
//q使用自定义的数据对象
CorrelationDataExt correlationData = new CorrelationDataExt();
correlationData.setId(messageid);
correlationData.setData(jmsg);
correlationData.setQueueName(CommonConstant.OrderToInventoryQueue);
correlationData.setExchangeName(CommonConstant.OrderToInventoryExchage);
correlationData.setRoutName(CommonConstant.OrderToInventoryRout);
rabbitTemplate.setEncoding("UTF-8");
//设置消息的请求头部参数，匿名内部类
MessagePostProcessor mpp = new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setMessageId(messageid);
//设置消息持久化
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}
};
try {
rabbitTemplate.convertAndSend(CommonConstant.OrderToInventoryExchage, CommonConstant.OrderToInventoryRout, jmsg,mpp);
}catch(Exception e) {
//q保存报错信息到DB
String key = CommonConstant.ReduceInventoryFailure;
jedisCluster.rpush(key, jmsg);
return CommonConstant.failure;
}
return messageid;
}消息监听
@Bean
public RabbitTemplate createRabbitTemplate(ConnectionFactory connectionFactory) {
RabbitTemplate rabbitTemplate = new RabbitTemplate();
rabbitTemplate.setConnectionFactory(connectionFactory);
// q设置开启Mandatory,才能触发回调函数,无论消息推送结果怎么样都强制调用回调函数
rabbitTemplate.setMandatory(true);
// 消息推送到server，但是在server里找不到交换机
rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
CorrelationDataExt correlationDataExt = (CorrelationDataExt) correlationData;
if (!ack) {
if (correlationData instanceof CorrelationDataExt) {
sendToRedis(correlationDataExt.getId(),correlationDataExt.getExchangeName(),correlationDataExt.getRoutName(),correlationDataExt.getData());
}
}else {
//判断之前是否有发送失败的，有的话删除
checkExistRedisWhenSendSuccess(correlationDataExt.getId());
}
}
});
// 消息推送到server，找到交换机了，但是没找到队列
rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
@Override
public void returnedMessage(Message message, int replyCode, String replyText, String exchange,
String routingKey) {
sendToRedis(message.getMessageProperties().getMessageId(),exchange,routingKey,new String(message.getBody()));
}
});
return rabbitTemplate;
}
/**
* 写入Redis做定时插入
* @param messageId
* @param exchange
* @param routkey
* @param body
*/
public void sendToRedis(String messageId,String exchange,String routkey,Object body) {
SendFailureMsg sfm = new SendFailureMsg();
redisConfig.getJedisCluster().getClusterNodes();
JedisCluster redis = redisConfig.getJedisCluster();
if (redis.exists(failureKey)) {
String FailureMessageJson = jedisCluster.get(failureKey);
ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
SendFailureMsg.class);
// q重新入队
if (messageOfOrderList.size() > 0) {
for (SendFailureMsg moo : messageOfOrderList) {
if (moo.getMessageId().equals(messageId)) {
String jvalue = CommonUtils.ObjectToJson(moo);
jedisCluster.lrem(failureKey, 1, jvalue);
moo.setFailureCount(moo.getFailureCount() + 1);
jedisCluster.rpush(failureKey, JSON.toJSONString(moo));
existInRedis = true;
break;
}
}
}
}
if (!existInRedis) {
sfm.setBody(body);
sfm.setExchange(exchange);
sfm.setRout(routkey);
sfm.setMessageId(messageId);
sfm.setFailureCount(0);
// q保存报错信息到DB
String key = CommonConstant.ReduceInventoryFailure;
jedisCluster.rpush(key, JSON.toJSONString(sfm));
}
}
/**
* 发送成功，则检查是否有失败过，如果有就将其删除
* @param messageId
*/
public void checkExistRedisWhenSendSuccess(String messageId) {
redisConfig.getJedisCluster().getClusterNodes();
JedisCluster redis = redisConfig.getJedisCluster();
if (redis.exists(failureKey)) {
String FailureMessageJson = jedisCluster.get(failureKey);
ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
SendFailureMsg.class);
// q重新入队
if (messageOfOrderList.size() > 0) {
for (SendFailureMsg moo : messageOfOrderList) {
if (moo.getMessageId().equals(messageId)) {
String jvalue = CommonUtils.ObjectToJson(moo);
jedisCluster.lrem(failureKey, 1, jvalue);
break;
}
}
}
}
}写个定时任务自动处理发送失败的消息
@Component
@EnableScheduling
@EnableAsync
public class MultithreadScheduleTask {
@Autowired
RabbitTemplate rabbitTemplate; // q使用RabbitTemplate,这提供了接收/发送等等方法
@Autowired
private JedisCluster jedisCluster;
@Autowired
private RedisConfig redisConfig;
@Async
@Scheduled(fixedDelay = 1000 * 60 * 30)
public void ReduceInventoryFailureMessage() throws InterruptedException {
System.out.println(
"第一个定时任务开始 : " + LocalDateTime.now().toLocalTime() + "\r\n线程 : " + Thread.currentThread().getName());
redisConfig.getJedisCluster().getClusterNodes();
JedisCluster redis = redisConfig.getJedisCluster();
if (redis.exists(CommonConstant.ReduceInventoryFailure)) {
String FailureMessageJson = jedisCluster.get(CommonConstant.ReduceInventoryFailure);
ArrayList<SendFailureMsg> messageOfOrderList = new ArrayList<SendFailureMsg>();
messageOfOrderList = (ArrayList<SendFailureMsg>) JSONArray.parseArray(FailureMessageJson,
SendFailureMsg.class);
// q重新入队
if (messageOfOrderList.size() > 0) {
for (SendFailureMsg moo : messageOfOrderList) {
try {
if (Integer.valueOf(moo.getFailureCount()) > 5) {
ResendDel(CommonConstant.ReduceInventoryFailure,moo);
} else {
MessagePostProcessor mpp = new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setMessageId(moo.getMessageId());
return message;
}
};
rabbitTemplate.convertAndSend(moo.getExchange(), moo.getRout(), moo.getBody(), mpp);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
}
/**
* 重新发送成功或失败次数超过5次则从缓存删除数据
*
* @param id
* @param moo
* @param reduceinventoryfailure
*/
public void ResendDel(String key, SendFailureMsg value) {
String jvalue = CommonUtils.ObjectToJson(value);
jedisCluster.lrem(key, 1, jvalue);
}
/**
* 重写发送
*
* @param id
*/
public void ResendFailed(String key, SendFailureMsg moo) {
MessagePostProcessor mpp = new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
message.getMessageProperties().setMessageId(moo.getMessageId());
return message;
}
};
rabbitTemplate.convertAndSend(moo.getExchange(), moo.getRout(), moo.getBody(), mpp);
}3.中间件挂了弄丢了消息
开启RabbitMQ队列持久化和消息持久化
3.1队列持久化
Bean创建队列
@Bean
public Queue DirectQueue() {
//第二个参数为true,队列持久化
return new Queue(CommonConstant.OrderToInventoryQueue, true);
}channel创建队列
Connection connection = connectionFactory.newConnection();
Channel channel = connection.createChannel();
//第二个参数：是否持久化；排他队列；自动删除
channel.queueDeclare(CommonConstant.OrderToInventoryQueue, true, false, false, null);
3.2消息持久化
设置消息头部信息deliverymode=2，deliverymode=1是消息不持久化
3.2.1springboot整合RabbitMQ的rabbitTemplate
MessageDeliveryMode.PERSISTENT 返回2
MessageDeliveryMode.NON_PERSISTENT 返回1
//设置消息的请求头部参数，匿名内部类
MessagePostProcessor mpp = new MessagePostProcessor() {
@Override
public Message postProcessMessage(Message message) throws AmqpException {
//设置消息持久化
message.getMessageProperties().setDeliveryMode(MessageDeliveryMode.PERSISTENT);
return message;
}
};
rabbitTemplate.convertAndSend(CommonConstant.OrderToInventoryExchage, CommonConstant.OrderToInventoryRout, jmsg,mpp);3.2.2channel发布持久化消息
MessageProperties.PERSISTENT_TEXT_PLAIN 返回2
MessageProperties.TEXT_PLAIN 返回1
//设置第三个参数deliverymode为2
channel.basicPublish("exchange", "routKey", MessageProperties.PERSISTENT_TEXT_PLAIN, messagebody.getBytes());
4.消费者弄丢了消息
消费者手动确认机制
container.setAcknowledgeMode(AcknowledgeMode.MANUAL); // RabbitMQ默认是自动确认，这里改为手动确认消息
Boolean excute = true;
try {
//消费者消费过程
result = inventoryReduceApi.reduceInventory(message, messageId);
} catch (Exception e) {
excute = false;
} finally {
if (excute) {
//手动确认
channel.basicAck(deliveryTag, true);
} else {
// 第二个参数是否requeue，true则重新入队列，否则丢弃或者进入死信队列
channel.basicReject(deliveryTag, false);
// 将消息放到队尾
channel.basicPublish(exchange, rout, MessageProperties.PERSISTENT_TEXT_PLAIN, message.getBytes());
}
}
扫一扫
1万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
