RabbitMQ
一、概述
1. MQ
消息队列(Message Queue)是在消息传输过程中保存消息的队列容器,用于分布式系统之间进行通信
2. 选型和对比
RabbitMQ | ActiveMQ | RocketMQ | Kafka | |
---|---|---|---|---|
公司/社区 | Rabbit | Apache | 阿里 | Apache |
开发语言 | Erlang | Java | Java | Scala&Java |
协议 | AMQP | OpenWire、AUTO、Stomp、MQTT | 自定义 | 自定义 |
单机吞吐量 | 万级 | 万级(最差) | 十万级 | 十万级 |
消息延迟 | 微妙级 | 毫秒级 | 毫秒级 | 毫秒以内 |
特性 | 并发能力很强,延时很低 | 老牌产品,文档较多 | MQ功能比较完备,扩展性佳 | 只支持主要的MQ功能,毕竟是为大数据领域准备的。 |
数据量小,并发高的选择RabbitMQ
数据量大,选择 RocketMQ 或 Kafka,需要日志采集功能选择 Kafka
3. 采用MQ的原因(MQ的使用场景)
3.1 解耦
**传统模式:**系统耦合度太强,系统A调用系统B和C,若要添加系统D,则需修改系统A的代码,十分麻烦。并且任何一个子系统出现了故障,都会导致系统A不可用
**中间件模式:**将消息写入消息队列中,需要消息的系统自己去订阅。则若添加新的子系统D,只需让其去队列中获取消息即可,不需要对系统A进行修改。若子系统出现故障,也可将消息暂时缓存在消息队列中,等故障处理完毕,再去获取消息即可,不影响总业务的处理
3.2 削峰
并发量较大时,同时访问数据库可能会出现问题,则需把处理能力之外的请求拒绝。使用中间件则可把多余的请求缓存到队列中,根据系统处理能力慢慢处理消息。
比如说一个订单系统最多能处理一万次订单,但如果有两万个订单就无法全部处理,只能限制订单超过一万后不允许用户下单。使用消息队列做缓冲,可以把订单缓存到队列,系统分批次拉取消息进行处理,可能有些用户下单十几秒后才收到下单成功的操作,但总比不能下单的体验要好。
3.3 异步
传统模式:非必要的业务逻辑以同步的方式运行,影响主业务的处理时间
中间件模式: 将消息写入消息队列,非必要的业务逻辑以异步的方式运行,加快响应速度
比如说下订单的操作,用户发送下订单的请求,订单系统只需50ms就可以处理完,但是订单系统还需要调用其他的服务比如短信邮件什么的,这些服务都会占用处理时间,最后可能200ms订单才响应成功。但是如果使用消息中间件的话,则用户发送订单请求,订单系统在调用其他服务的时候,把订单消息发送到消息队列中,就可以直接响应下单成功,其他服务在拉取到消息后慢慢执行就可以了,并不影响订单成功的响应
二、安装
1. 下载
或直接下载提供 安装包
2. 安装Erlang
-
上传安装包
-
执行命令安装
rpm -ivh esl-erlang-17.3-1.x86_64.rpm --force --nodeps rpm -ivh esl-erlang_17.3-1~centos~6_amd64.rpm --force --nodeps rpm -ivh esl-erlang-compat-R14B-1.el6.noarch.rpm --force --nodeps
3. 安装RabbitMQ
-
上传安装包
-
安装命令
rpm -ivh rabbitmq-server-3.4.1-1.noarch.rpm
-
启动和停止
service rabbitmq-server start service rabbitmq-server stop service rabbitmq-server restart # 查看状态 service rabbitmq-server status
-
开机自启
chkconfig rabbitmq-server on
-
开放 15672 端口
# 开放15672端口 firewall-cmd --zone=public --add-port=15672/tcp --permanent # 重启防火墙 firewall-cmd --reload # 查看开放的端口 firewall-cmd --zone=public --list-ports # 或者直接开机禁用防火墙 systemctl disable firewalld
4. UI界面管理工具
4.1 开启web界面管理工具
rabbitmq-plugins enable rabbitmq_management
service rabbitmq-server restart
4.2 创建账户
创建用户:admin,密码:admin
-
创建账户
rabbitmqctl add_user admin admin
-
设置用户角色
rabbitmqctl set_user_tags admin administrator
-
设置用户权限
rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"
-
查看当前用户和角色(在服务开启状态下查看)
rabbitmqctl list_users
-
访问:http://192.168.40.138:15672/
三、五种消息模式
1. 简单消息模式
1.1 特征
- P:生产者,发送消息
- C:消费者,处理消费消息
- 队列:传递消息的临时缓存空间,许多生产者可以发送消息到一个队列,许多消费者可以尝试从一个队列接收数据
1.2 java 端
RabbitMQUtil
public class RabbitMQUtil {
//获取连接
public static Connection getConnection() throws IOException, TimeoutException {
ConnectionFactory factory = new ConnectionFactory();
factory.setHost("192.168.40.138");
factory.setPort(5672);
factory.setUsername("admin");
factory.setPassword("1111");
factory.setVirtualHost("/"); //虚拟主机
return factory.newConnection();
}
}
生产者
//简单消息模式消息发送者
public class SimpleSend {
private static final String QUEUE_NAME = "simple-queue";
public static void main(String[] args) throws IOException, TimeoutException {
//建立连接
Connection connection = RabbitMQUtil.getConnection();
//创建通道,是轻量级的connection
Channel channel = connection.createChannel();
//声明队列,队列可以暂时存储消息
channel.queueDeclare(QUEUE_NAME, false, false, false , null);
for(int i=0;i<10;i++){
//声明消息
String msg = "简单模式"+i;
//推送消息
channel.basicPublish("", QUEUE_NAME, null, msg.getBytes(StandardCharsets.UTF_8));
}
//关闭通道连接
channel.close();
connection.close();
}
}
消费者
//简单消息消费者
public class SimpleConsumer {
private static final String QUEUE_NAME = "simple-queue";
public static void main(String[] args) throws IOException, TimeoutException {
//建立连接
Connection connection = RabbitMQUtil.getConnection();
//创建通道,是轻量级的connection
Channel channel = connection.createChannel();
//声明队列,队列可以暂时存储消息
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
DefaultConsumer consumer = new DefaultConsumer(channel) {
//回调函数处理消息
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
//消费者处理消息的逻辑
String msg = new String(body);
System.out.println(msg);
if(msg.contains("5")){
//一次性拿到10条消息,若消息4处理失败,则消息0-3会从队列删除,剩余6条消息仍保留在队列
int i = 1/0;
}
/*
* 手动ack,视情况决定自动还是手动ack
* long deliveryTag: 当前消息索引
* boolean multiple:是否处理多条消息, true则一次性ack所有小于deliveryTag的消息
*/
channel.basicAck(envelope.getDeliveryTag(), false);
}
};
/* 消费者拉取消息
* String queue: 队列
* boolean autoAck: 是否自动ACK,只有ACK后消息才会从队列删除
* Consumer callback: 收到消息后执行其回调函数处理消息
*/
channel.basicConsume(QUEUE_NAME, false, consumer);
}
结果
2. Work-工作模型
- 共同处理:一个消费者处理效率低下,可创建多个消费者共同处理消息,但一个消息只能被一个消费者处理
- 能者多劳:设置为多个消费者时,消息会被平均分配给各个消费者,如两个消费者,则奇数消息给消费者1,偶数消息给消费者2, 一次性分配完毕,消费者拿到属于自己的所有消息再慢慢消费。但消费者的处理效率不同,平均分配可能导致有的消息挤压,有的处理完空闲。此时不再一次性分配所有消息,令其一次分配一条消息,消费者处理完再过来进行分配,则性能好的消费者请求频繁分配的消息会更多,性能低的分配的消息数量也低
- 设置每次处理一条消息:channel.basicQos(1);
- 设置提交为手动 ack
生产者
public class WorkSend {
private static final String QUEUE_NAME = "work-queue";
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
for (int i = 0; i < 50; i++) {
String msg = "工作模式"+i;
channel.basicPublish("", QUEUE_NAME, null, msg.getBytes());
}
channel.close();
connection.close();
}
}
消费者1
public class WorkConsumer1 {
private static final String QUEUE_NAME = "work-queue";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//每次处理一条消息,能者多劳
channel.basicQos(1);
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println(new String(body));
//手动ack
channel.basicAck(envelope.getDeliveryTag(), false);
}
};
//自动ack
channel.basicConsume(QUEUE_NAME, false, consumer);
}
}
消费者2
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//每次处理一条消息,能者多劳
channel.basicQos(1);
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println(new String(body));
//每次睡一秒,模拟处理效率低
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//手动ack
channel.basicAck(envelope.getDeliveryTag(), false);
}
};
channel.basicConsume(QUEUE_NAME, false, consumer);
}
}
结果
3. Fanout-广播模型
生产者
生产者不再创建队列,而是创建交换机,由交换机决定将消息分发到哪些队列中,fanout模式是分发到所有队列,每个队列都有此消息,则可被多次消费
//广播模式
public class FanoutSend {
private static final String EXCHANGE_NAME = "fanout-exchange";
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
//不再声明队列,而是声明交换机
channel.exchangeDeclare(EXCHANGE_NAME, "fanout");
String msg = "广播模式";
channel.basicPublish(EXCHANGE_NAME, "", null, msg.getBytes());
channel.close();
connection.close();
}
}
消费者1
public class FanoutConsumer1 {
private static final String EXCHANGE_NAME = "fanout-exchange";
//队列1
private static final String QUEUE_NAME = "fanout-queue-1";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-1"+new String(body));
}
};
//自动ack
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
消费者2
public class FanoutConsumer2 {
private static final String EXCHANGE_NAME = "fanout-exchange";
//队列1
private static final String QUEUE_NAME = "fanout-queue-2";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-2"+new String(body));
}
};
//自动ack
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
结果
4. Direct-定向模型
交换机不再将消息发送给所有队列,而是发送给指定队列。
生产者向交换机发送消息时指定 routing key,消费者的队列与交换机绑定时也会携带一些 routing key,则交换机会向那些携带 routing key 中包含生产者 routing key 的队列发送消息
P:生产者,向Exchange发送消息,发送消息时,会指定一个routing key
X:Exchange(交换机),接收生产者的消息,然后把消息递交给 与routing key完全匹配的队列
C1:消费者,其所在队列指定了需要routing key 为 error 的消息
C2:消费者,其所在队列指定了需要routing key 为 info、error、warning 的消息
生产者
//定向模式
public class DirectSend {
private static final String EXCHANGE_NAME = "direct-exchange";
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "direct");
String msg = "定向模式-error";
//指定 routing key 为 info
channel.basicPublish(EXCHANGE_NAME, "error", null, msg.getBytes());
channel.close();
connection.close();
}
}
消费者1
public class DirectConsumer1 {
private static final String EXCHANGE_NAME = "direct-exchange";
//队列1
private static final String QUEUE_NAME = "direct-queue-1";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//绑定交换机时指定 routing key 为 info, error, warning
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "info");
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "error");
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "warning");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-1"+new String(body));
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
消费者2
public class DirectConsumer2 {
private static final String EXCHANGE_NAME = "direct-exchange";
//队列1
private static final String QUEUE_NAME = "direct-queue-2";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//绑定交换机时指定 routing key 为 info, error, warning
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "error");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-1"+new String(body));
}
};
//自动ack
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
结果:只有消费者1能收到消息
若生产者将 routing key 改为 error
String msg = "定向模式-error";
//指定 routing key 为 error
channel.basicPublish(EXCHANGE_NAME, "error", null, msg.getBytes());
结果:两消费者都能收到
5. Topic-主题模型
生产者
//定向模式
public class TopicSend {
private static final String EXCHANGE_NAME = "topic-exchange";
public static void main(String[] args) throws IOException, TimeoutException, InterruptedException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.exchangeDeclare(EXCHANGE_NAME, "topic");
String msg = "主题模式-routing key: lazy.orange.abc";
//指定 routing key 为 abc.eee.rabbit
channel.basicPublish(EXCHANGE_NAME, "lazy.orange.abc", null, msg.getBytes());
channel.close();
connection.close();
}
}
消费者1
public class TopicConsumer1 {
private static final String EXCHANGE_NAME = "topic-exchange";
//队列1
private static final String QUEUE_NAME = "topic-queue-1";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//绑定交换机时指定 routing key 为 *.orange.*
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "*.orange.*");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-1"+new String(body));
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
消费者2
public class TopicConsumer2 {
private static final String EXCHANGE_NAME = "topic-exchange";
//队列1
private static final String QUEUE_NAME = "topic-queue-2";
public static void main(String[] args) throws IOException, TimeoutException {
Connection connection = RabbitMQUtil.getConnection();
Channel channel = connection.createChannel();
channel.queueDeclare(QUEUE_NAME, false, false, false, null);
//绑定交换机时指定 routing key 为 *.*.rabbit 和 lazy.#
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "*.*.rabbit");
channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "lazy.#");
DefaultConsumer consumer = new DefaultConsumer(channel){
@Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
System.out.println("consumer-1"+new String(body));
}
};
channel.basicConsume(QUEUE_NAME, true, consumer);
}
}
结果:队列1 不匹配,队列2 的 *.*.rabbit
与之匹配
修改生产者的 routing key 为 lazy.orange.abc
String msg = "主题模式-routing key: lazy.orange.abc";
//指定 routing key 为 abc.eee.rabbit
channel.basicPublish(EXCHANGE_NAME, "lazy.orange.abc", null, msg.getBytes());
结果:队列1 的 *.orange.*
和队列2 的lazy.#
都与之匹配
四、持久化
消息丢失:
- 消息丢失可以分为两种丢失
- 一种是消息队列服务器宕机或重启导致消息丢失
- 一种是消费者拉取消息后使用自动提交ack,但提交完ack消息队列把此消息删除后,消费者处理此消息失败,则消息将会丢失
防止消息丢失:
-
持久化:针对消息队列服务器宕机或重启导致的丢失,将
消息、队列和交换机
三部分都持久化保存到本地-
交换机
//(交换机名称,消息模式,开启交换机持久化); channel.exchangeDeclare(EXCHANGE_NAME, "topic", true);
-
队列
//(队列名,开启队列持久化,false,null) channel.queueDeclare(QUEUE_NAME, true, false, false, null);
-
消息
//(交换机名,routing-key,开启消息持久化,消息内容) channel.basicPublish(EXCHANGE_NAME, "lazy.orange.abc", MessageProperties.PERSISTENT_TEXT_PLAIN, msg.getBytes());
-
-
手动ack:针对消费者处理消息失败导致的丢失,将自动ack改为在处理消息成功后手动ack即可
- 手动ack,分别测试持久化和非持久化:
- 发送50条消息
- 消费者收到一条消息sleep1秒钟,收到前几条消息后立即关闭
- 重启RabbitMQ观察消息是否丢失
- 发现持久化后消息会继续刚才往下接收
五、Spring AMQP
Spring-amqp是对AMQP协议的抽象实现,而spring-rabbit 是对协议的具体实现,也是目前的唯一实现。底层使用的就是RabbitMQ。
springboot整合
1. 导入依赖
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
</dependencies>
2. yml配置文件
#主机,端口,用户名,密码,虚拟端口
spring:
rabbitmq:
host: 192.168.40.138
port: 5672
username: admin
password: 1111
virtual-host: /
3. 创建消息接收者
@Component
public class AmqpConsumer {
/**
* 监听者接收消息三要素:
* 1、queue
* 2、exchange
* 3、routing key
*/
@RabbitListener(bindings = @QueueBinding(
value = @Queue(value="springboot_queue",durable = "true"), // durable:队列是否持久化,默认true可不写
exchange = @Exchange(value="springboot_exchange",type= ExchangeTypes.TOPIC),
key= {"*.*"}
))
public void listen(String msg){
System.out.println("接收到消息:" + msg);
}
}
4. 测试类发送消息
@SpringBootTest
public class AmqpSendTest {
@Autowired
private AmqpTemplate amqpTemplate;
@Test
void sendTest() throws InterruptedException {
String msg = "springboot-整合";
amqpTemplate.convertAndSend("springboot_exchange", "a.b", msg);
Thread.sleep(3000);
}
}
5. 测试
启动测试方法
6. 手动ack
配置文件修改
#主机,端口,用户名,密码,虚拟端口
spring:
rabbitmq:
host: 192.168.40.138
port: 5672
username: admin
password: 1111
virtual-host: /
# 设置手动ack则添加此配置,默认自动ack
listener:
#设置前两种work消息类型手动ack
simple:
acknowledge-mode: manual
#设置后三种订阅模式手动ack
direct:
acknowledge-mode: manual
消费者监听方法修改
监听方法添加Channel channel, Message message两个参数
@Component
public class AmqpConsumer {
@RabbitListener(bindings = @QueueBinding(
value = @Queue(value="springboot_queue",durable = "true"),
exchange = @Exchange(value="springboot_exchange",type= ExchangeTypes.TOPIC),
key= {"*.*"}
))
public void listen(String msg, Channel channel, Message message){
System.out.println("接收到消息:" + msg);
//手动ack
channel.basicAck(message.getMessageProperties().getDeliveryTag(),false);
}
}