https://www.imooc.com/article/49814
参考地址:
https://www.imooc.com/t/2726237
谈到消息的可靠性投递,无法避免的,在实际的工作中会经常碰到,比如一些核心业务需要保障消息不丢失,接下来我们看一个可靠性投递的流程图,说明可靠性投递的概念:
-
Step 1: 首先把消息信息(业务数据)存储到数据库中,紧接着,我们再把这个消息记录也存储到一张消息记录表里(或者另外一个同源数据库的消息记录表)
-
Step 2:发送消息到MQ Broker节点(采用confirm方式发送,会有异步的返回结果)
-
Step 3、4:生产者端接受MQ Broker节点返回的Confirm确认消息结果,然后进行更新消息记录表里的消息状态。比如默认Status = 0 当收到消息确认成功后,更新为1即可!
-
Step 5:但是在消息确认这个过程中可能由于网络闪断、MQ Broker端异常等原因导致 回送消息失败或者异常。这个时候就需要发送方(生产者)对消息进行可靠性投递了,保障消息不丢失,100%的投递成功!(有一种极限情况是闪断,Broker返回的成功确认消息,但是生产端由于网络闪断没收到,这个时候重新投递可能会造成消息重复,需要消费端去做幂等处理)所以我们需要有一个定时任务,(比如每5分钟拉取一下处于中间状态的消息,当然这个消息可以设置一个超时时间,比如超过1分钟 Status = 0 ,也就说明了1分钟这个时间窗口内,我们的消息没有被确认,那么会被定时任务拉取出来)
-
Step 6:接下来我们把中间状态的消息进行重新投递 retry send,继续发送消息到MQ ,当然也可能有多种原因导致发送失败
-
Step 7:我们可以采用设置最大努力尝试次数,比如投递了3次,还是失败,那么我们可以将最终状态设置为Status = 2 ,最后 交由人工解决处理此类问题(或者把消息转储到失败表中)。
接下来,我们使用SpringBoot2.x 实现这一可靠性投递策略:
废话不多说,直接上代码:
-
数据库库表结构:订单表和消息记录表
-- 表 order 订单结构 CREATE TABLE IF NOT EXISTS `t_order` ( `id` varchar(128) NOT NULL, -- 订单ID `name` varchar(128), -- 订单名称 其他业务熟悉忽略 `message_id` varchar(128) NOT NULL, -- 消息唯一ID PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8; -- 表 broker_message_log 消息记录结构 CREATE TABLE IF NOT EXISTS `broker_message_log` ( `message_id` varchar(128) NOT NULL, -- 消息唯一ID `message` varchar(4000) DEFAULT NULL, -- 消息内容 `try_count` int(4) DEFAULT '0', -- 重试次数 `status` varchar(10) DEFAULT '', -- 消息投递状态 0 投递中 1 投递成功 2 投递失败 `next_retry` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', -- 下一次重试时间 或 超时时间 `create_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', -- 创建时间 `update_time` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00', -- 更新时间 PRIMARY KEY (`message_id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-
整合SpringBoot 实现生产端代码如下:pom.xml配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
|
-
application.properties配置:
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
spring.rabbitmq.addresses=
192.168
.
11.76
:
5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest
spring.rabbitmq.virtual-host=/
spring.rabbitmq.connection-timeout=
15000
spring.rabbitmq.publisher-confirms=
true
spring.rabbitmq.publisher-returns=
true
spring.rabbitmq.template.mandatory=
true
server.servlet.context-path=/
server.port=
8001
spring.http.encoding.charset=UTF-
8
spring.jackson.date-format=yyyy-MM-dd HH:mm:ss
spring.jackson.time-zone=GMT+
8
spring.jackson.
default
-property-inclusion=NON_NULL
spring.datasource.type=com.alibaba.druid.pool.DruidDataSource
spring.datasource.url=jdbc:mysql:
//localhost:3306/test?characterEncoding=UTF-8&autoReconnect=true&zeroDateTimeBehavior=convertToNull&useUnicode=true
spring.datasource.driver-
class
-name=com.mysql.jdbc.Driver
spring.datasource.username=root
spring.datasource.password=root
mybatis.type-aliases-
package
=com.bfxy.springboot
mybatis.mapper-locations=classpath:com/bfxy/springboot/mapping/*.xml
logging.level.tk.mybatis=TRACE
-
数据源druid.properties配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
|
-
实体对象:
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package
com.bfxy.springboot.entity;
import
java.io.Serializable;
public
class
Order
implements
Serializable {
private
static
final
long
serialVersionUID = 9111357402963030257L;
private
String id;
private
String name;
private
String messageId;
public
String getId() {
return
id;
}
public
void
setId(String id) {
this
.id = id ==
null
?
null
: id.trim();
}
public
String getName() {
return
name;
}
public
void
setName(String name) {
this
.name = name ==
null
?
null
: name.trim();
}
public
String getMessageId() {
return
messageId;
}
public
void
setMessageId(String messageId) {
this
.messageId = messageId ==
null
?
null
: messageId.trim();
}
}
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
package
com.bfxy.springboot.entity;
import
java.util.Date;
public
class
BrokerMessageLog {
private
String messageId;
private
String message;
private
Integer tryCount;
private
String status;
private
Date nextRetry;
private
Date createTime;
private
Date updateTime;
public
String getMessageId() {
return
messageId;
}
public
void
setMessageId(String messageId) {
this
.messageId = messageId ==
null
?
null
: messageId.trim();
}
public
String getMessage() {
return
message;
}
public
void
setMessage(String message) {
this
.message = message ==
null
?
null
: message.trim();
}
public
Integer getTryCount() {
return
tryCount;
}
public
void
setTryCount(Integer tryCount) {
this
.tryCount = tryCount;
}
public
String getStatus() {
return
status;
}
public
void
setStatus(String status) {
this
.status = status ==
null
?
null
: status.trim();
}
public
Date getNextRetry() {
return
nextRetry;
}
public
void
setNextRetry(Date nextRetry) {
this
.nextRetry = nextRetry;
}
public
Date getCreateTime() {
return
createTime;
}
public
void
setCreateTime(Date createTime) {
this
.createTime = createTime;
}
public
Date getUpdateTime() {
return
updateTime;
}
public
void
setUpdateTime(Date updateTime) {
this
.updateTime = updateTime;
}
}
-
数据库连接池代码:
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
package
com.bfxy.springboot.config.database;
import
java.sql.SQLException;
import
javax.sql.DataSource;
import
org.slf4j.Logger;
import
org.slf4j.LoggerFactory;
import
org.springframework.beans.factory.annotation.Autowired;
import
org.springframework.context.annotation.Bean;
import
org.springframework.context.annotation.Configuration;
import
org.springframework.context.support.PropertySourcesPlaceholderConfigurer;
import
org.springframework.jdbc.datasource.DataSourceTransactionManager;
import
org.springframework.transaction.PlatformTransactionManager;
import
org.springframework.transaction.annotation.EnableTransactionManagement;
import
com.alibaba.druid.pool.DruidDataSource;
@Configuration
@EnableTransactionManagement
public
class
DruidDataSourceConfig {
private
static
Logger logger = LoggerFactory.getLogger(DruidDataSourceConfig.
class
);
@Autowired
private
DruidDataSourceSettings druidSettings;
public
static
String DRIVER_CLASSNAME ;
@Bean
public
static
PropertySourcesPlaceholderConfigurer propertyConfigure(){
return
new
PropertySourcesPlaceholderConfigurer();
}
@Bean
public
DataSource dataSource()
throws
SQLException {
DruidDataSource ds =
new
DruidDataSource();
ds.setDriverClassName(druidSettings.getDriverClassName());
DRIVER_CLASSNAME = druidSettings.getDriverClassName();
ds.setUrl(druidSettings.getUrl());
ds.setUsername(druidSettings.getUsername());
ds.setPassword(druidSettings.getPassword());
ds.setInitialSize(druidSettings.getInitialSize());
ds.setMinIdle(druidSettings.getMinIdle());
ds.setMaxActive(druidSettings.getMaxActive());
ds.setTimeBetweenEvictionRunsMillis(druidSettings.getTimeBetweenEvictionRunsMillis());
ds.setMinEvictableIdleTimeMillis(druidSettings.getMinEvictableIdleTimeMillis());
ds.setValidationQuery(druidSettings.getValidationQuery());
ds.setTestWhileIdle(druidSettings.isTestWhileIdle());
ds.setTestOnBorrow(druidSettings.isTestOnBorrow());
ds.setTestOnReturn(druidSettings.isTestOnReturn());
ds.setPoolPreparedStatements(druidSettings.isPoolPreparedStatements());
ds.setMaxPoolPreparedStatementPerConnectionSize(druidSettings.getMaxPoolPreparedStatementPerConnectionSize());
ds.setFilters(druidSettings.getFilters());
ds.setConnectionProperties(druidSettings.getConnectionProperties());
logger.info(
" druid datasource config : {} "
, ds);
return
ds;
}
@Bean
public
PlatformTransactionManager transactionManager()
throws
Exception {
DataSourceTransactionManager txManager =
new
DataSourceTransactionManager();
txManager.setDataSource(dataSource());
return
txManager;
}
}
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
|
-
定时任务配置代码:
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package
com.bfxy.springboot.config.task;
import
java.util.concurrent.Executor;
import
java.util.concurrent.Executors;
import
org.springframework.context.annotation.Bean;
import
org.springframework.context.annotation.Configuration;
import
org.springframework.scheduling.annotation.EnableScheduling;
import
org.springframework.scheduling.annotation.SchedulingConfigurer;
import
org.springframework.scheduling.config.ScheduledTaskRegistrar;
@Configuration
@EnableScheduling
public
class
TaskSchedulerConfig
implements
SchedulingConfigurer {
@Override
public
void
configureTasks(ScheduledTaskRegistrar taskRegistrar) {
taskRegistrar.setScheduler(taskScheduler());
}
@Bean
(destroyMethod=
"shutdown"
)
public
Executor taskScheduler(){
return
Executors.newScheduledThreadPool(
100
);
}
}
-
常量类:
1 2 3 4 5 6 7 8 9 10 11 12 |
|
-
消息记录表核心业务:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
|
-
对应的SQL代码:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
|
-
核心发送代码:orderService
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
package
com.bfxy.springboot.service;
import
java.util.Date;
import
org.apache.commons.lang3.time.DateUtils;
import
org.springframework.beans.factory.annotation.Autowired;
import
org.springframework.stereotype.Service;
import
com.bfxy.springboot.constant.Constants;
import
com.bfxy.springboot.entity.BrokerMessageLog;
import
com.bfxy.springboot.entity.Order;
import
com.bfxy.springboot.mapper.BrokerMessageLogMapper;
import
com.bfxy.springboot.mapper.OrderMapper;
import
com.bfxy.springboot.producer.RabbitOrderSender;
import
com.bfxy.springboot.utils.FastJsonConvertUtil;
@Service
public
class
OrderService {
@Autowired
private
OrderMapper orderMapper;
@Autowired
private
BrokerMessageLogMapper brokerMessageLogMapper;
@Autowired
private
RabbitOrderSender rabbitOrderSender;
public
void
createOrder(Order order)
throws
Exception {
// 使用当前时间当做订单创建时间(为了模拟一下简化)
Date orderTime =
new
Date();
// 插入业务数据
orderMapper.insert(order);
// 插入消息记录表数据
BrokerMessageLog brokerMessageLog =
new
BrokerMessageLog();
// 消息唯一ID
brokerMessageLog.setMessageId(order.getMessageId());
// 保存消息整体 转为JSON 格式存储入库
brokerMessageLog.setMessage(FastJsonConvertUtil.convertObjectToJSON(order));
// 设置消息状态为0 表示发送中
brokerMessageLog.setStatus(
"0"
);
// 设置消息未确认超时时间窗口为 一分钟
brokerMessageLog.setNextRetry(DateUtils.addMinutes(orderTime, Constants.ORDER_TIMEOUT));
brokerMessageLog.setCreateTime(
new
Date());
brokerMessageLog.setUpdateTime(
new
Date());
brokerMessageLogMapper.insert(brokerMessageLog);
// 发送消息
rabbitOrderSender.sendOrder(order);
}
}
-
MQ消息发送核心代码:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
|
-
消息重试、最大努力尝试策略(定时任务):
-
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
package
com.bfxy.springboot.task;
import
java.util.Date;
import
java.util.List;
import
org.springframework.beans.factory.annotation.Autowired;
import
org.springframework.scheduling.annotation.Scheduled;
import
org.springframework.stereotype.Component;
import
com.bfxy.springboot.constant.Constants;
import
com.bfxy.springboot.entity.BrokerMessageLog;
import
com.bfxy.springboot.entity.Order;
import
com.bfxy.springboot.mapper.BrokerMessageLogMapper;
import
com.bfxy.springboot.producer.RabbitOrderSender;
import
com.bfxy.springboot.utils.FastJsonConvertUtil;
@Component
public
class
RetryMessageTasker {
@Autowired
private
RabbitOrderSender rabbitOrderSender;
@Autowired
private
BrokerMessageLogMapper brokerMessageLogMapper;
@Scheduled
(initialDelay =
5000
, fixedDelay =
10000
)
public
void
reSend(){
//pull status = 0 and timeout message
List<BrokerMessageLog> list = brokerMessageLogMapper.query4StatusAndTimeoutMessage();
list.forEach(messageLog -> {
if
(messageLog.getTryCount() >=
3
){
//update fail message
brokerMessageLogMapper.changeBrokerMessageLogStatus(messageLog.getMessageId(), Constants.ORDER_SEND_FAILURE,
new
Date());
}
else
{
// resend
brokerMessageLogMapper.update4ReSend(messageLog.getMessageId(),
new
Date());
Order reSendOrder = FastJsonConvertUtil.convertJSONToObject(messageLog.getMessage(), Order.
class
);
try
{
rabbitOrderSender.sendOrder(reSendOrder);
}
catch
(Exception e) {
e.printStackTrace();
System.err.println(
"-----------异常处理-----------"
);
}
}
});
}
}
-
1
测试发送订单:
-
代码如下:
1 2 3 4 5 6 7 8 9 10 11 |
|
-
监控台查看消息:
-
发送成功! 现在测试 发送订单并且入库(业务库和消息记录库)
1 2 3 4 5 6 7 8 9 10 11 |
|
-
发送成功 并且入库OK:业务表 和 消息记录表均有数据 且status状态=1 为成功!
-
业务表:
-
消息记录表:
-
测试失败情况:修改路由KEY为 无法路由即可!
-
这样消息就算失败的情况了。然后ACK的时候就会走异常处理,消息记录表如下:
-
最后我们测试重试策略:直接启动生产者应用,开启定时任务,重试几次后,库表信息变化如下:
-
最终重试3次 失败结果更新 status = 2
RabbitMQ消息中间件技术精讲:RabbitMQ是目前主流的消息中间件,非常适用于高并发环境。各大互联网公司都在使用的MQ技术,晋级技术骨干、团队核心的必备技术!
作者:阿神_
链接:https://www.imooc.com/article/49814
来源:慕课网
本文原创发布于慕课网 ,转载请注明出处,谢谢合作