MQ、AMQP、RabbitMQ

MQ作用:

(1)异步通信:客户端发出消息到服务端就返回了。服务端消费者会消费队列中的消息。整个过程不是阻塞式的请求/响应模式。如:转账时发出转账请求然后页面显示处理中,过了一段时间,在消息队列中等待处理转账的消费者处理结束,客户端无需持续等待。
(2)系统解耦:上游系统不再关心下游系统。如:订单系统的消息会进入消息队列,库存系统、支付系统、通知系统会自己消费消息,而不是订单系统直接将消息发给库存系统、支付系统、通知系统。
(3)流量消峰:流量控制。如:消息流量变大,直接添加一个消费者消费消息队列中的消息就行,生产者无需感知消费者。

带来的问题:

(1)系统的复杂度增加 :需要了解MQ,增加了开发难度。

(2)运维的成本增加:部署和管理MQ。

(3)应用之间数据的一致性风险增加:部分应用的消费者消费消息失败,造成数据不一致。

(4)系统的可用性降低:加长了消息传递路径,数据丢失可能性加大。MQ服务器宕机,无法进行消息传递。

AMQP(Advanced Message Queuing Protocol,高级消息队列协议): 

统一MQ标准,避免消息收发不兼容。

在服务器中,三个主要功能模块连接成一个处理链完成预期的功能:

“exchange”接收发布应用程序发送的消息,并根据一定的规则将这些消息路由到“消息队列”。

“message queue”存储消息,直到这些消息被消费者安全处理完为止。

“binding”定义了exchange和message queue之间的关联,提供路由规则。

生产者发送消息时的key是routingKey;队列和交换机绑定时的key是bindingKey。

https://baike.baidu.com/item/AMQP/8354716?fr=aladdin

RabbitMQ的Exchange(交换机):

(1)Direct(直连)交换机: 使用明确的绑定键。适用于业务目的明确的场景。如:员工入职的消息对应员工入职的消息队列。
(2)Top(主题)交换机:使用支持通配符的绑定键。适用于根据业务主题过滤消息的场景。如:
Binding Key(绑定键,RabbitMQ中routingKey和bindingKey都写成了routingKey)的通配符中:
*表示一个word
#表示0个、一个或多个word
如:senior.netty、junior.jvm
(3)Fanout(广播)Exchange:无需绑定键,适合通用业务消息。如:产品系统的消息被其他所有系统(推广、销售、维护等系统)对应的队列感知。

        <dependency>
            <groupId>com.rabbitmq</groupId>
            <artifactId>amqp-client</artifactId>
            <version>5.6.0</version>
        </dependency>
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("127.0.0.1");
        factory.setPort(5672);
        factory.setVirtualHost("/");
        factory.setUsername("hao");
        factory.setPassword("123");

        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();
        String msg = "Hello world, Rabbit MQ";

        // 发送消息到交换机
        // String exchange, String routingKey, BasicProperties props, byte[] body
        channel.basicPublish(EXCHANGE_NAME, ROUTING_KEY, null, "Hello Rabbit MQ".getBytes());

        channel.close();
        conn.close();
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost("127.0.0.1");
        factory.setPort(5672);
        factory.setVirtualHost("/");
        factory.setUsername("hao");
        factory.setPassword("123");
		
        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();
        // 交换机
        // String exchange, String type, boolean durable, boolean autoDelete, Map<String, Object> arguments
        channel.exchangeDeclare(EXCHANGE_NAME,"direct",false, false, null);

        // 队列
        // String queue, boolean durable, boolean exclusive, boolean autoDelete, Map<String, Object> arguments
        channel.queueDeclare(QUEUE_NAME, false, false, false, null);
        System.out.println(" Waiting for message....");

        // 交换机和队列队列
        channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, ROUTING_KEY);

        // 消费者
        Consumer consumer = new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
                                       byte[] body) throws IOException {
                String msg = new String(body, "UTF-8");
                System.out.println("Received message : '" + msg + "'");
                System.out.println("consumerTag : " + consumerTag );
                System.out.println("deliveryTag : " + envelope.getDeliveryTag() );
            }
        };

        // 监听消息到达队列
        // String queue, boolean autoAck, Consumer callback
        channel.basicConsume(QUEUE_NAME, true, consumer);

Spring-RabbitMQ: 

        <dependency>
            <groupId>org.springframework.amqp</groupId>
            <artifactId>spring-rabbit</artifactId>
            <version>1.3.5.RELEASE</version>
        </dependency>

配置RabbitMQ服务器、交换机、队列、消费者、生产者、 消息监听(消息到达队列,通知消费者)rabbitMQ.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:rabbit="http://www.springframework.org/schema/rabbit"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
     http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
     http://www.springframework.org/schema/rabbit
     http://www.springframework.org/schema/rabbit/spring-rabbit-1.2.xsd">

    <!-- rabbitServer -->
    <rabbit:connection-factory id="connectionFactory" virtual-host="/" username="hao" password="123" host="127.0.0.1" port="5672" />
    <rabbit:admin id="connectAdmin" connection-factory="connectionFactory" />

    <!-- 直连交换机 -->
    <rabbit:direct-exchange name="DIRECT_EXCHANGE" durable="true" auto-delete="false" declared-by="connectAdmin">
        <rabbit:bindings>
            <rabbit:binding queue="SECOND_QUEUE" key="SecondKey"></rabbit:binding>
        </rabbit:bindings>
    </rabbit:direct-exchange>
    <rabbit:direct-exchange name="DIRECT_EXCHANGE" durable="true" auto-delete="false" declared-by="connectAdmin">
        <rabbit:bindings>
            <rabbit:binding queue="FIRST_QUEUE" key="FirstKey">
            </rabbit:binding>
        </rabbit:bindings>
    </rabbit:direct-exchange>

    <!-- topic交换机 -->
    <rabbit:topic-exchange name="TOPIC_EXCHANGE" durable="true" auto-delete="false" declared-by="connectAdmin">
        <rabbit:bindings>
            <rabbit:binding queue="THIRD_QUEUE" pattern="#.Third.#"></rabbit:binding>
        </rabbit:bindings>
    </rabbit:topic-exchange>

    <!-- 广播交换机 -->
    <rabbit:fanout-exchange name="FANOUT_EXCHANGE" auto-delete="false" durable="true" declared-by="connectAdmin" >
        <rabbit:bindings>
            <rabbit:binding queue="FIRST_QUEUE"></rabbit:binding>
            <rabbit:binding queue="FOURTH_QUEUE"></rabbit:binding>
        </rabbit:bindings>
    </rabbit:fanout-exchange>

    <!-- 消息发送到交换机 -->
    <rabbit:template id="amqpTemplate2" connection-factory="connectionFactory" exchange="TOPIC_EXCHANGE" />
    <rabbit:template id="amqpTemplate" connection-factory="connectionFactory" exchange="DIRECT_EXCHANGE" />

    <!-- 消费者 -->
    <bean id="receiverThird" class="com.gupaoedu.consumer.ThirdConsumer"></bean>
    <bean id="receiverFourth" class="com.gupaoedu.consumer.FourthConsumer"></bean>
    <bean id="receiverSecond" class="com.gupaoedu.consumer.SecondConsumer"></bean>
    <bean id="messageReceiver" class="com.gupaoedu.consumer.FirstConsumer"></bean>

    <!--队列 -->
    <rabbit:queue name="FOURTH_QUEUE" durable="true" auto-delete="false" exclusive="false" declared-by="connectAdmin" />
    <rabbit:queue name="THIRD_QUEUE" durable="true" auto-delete="false" exclusive="false" declared-by="connectAdmin" />
    <rabbit:queue name="MY_SECOND_QUEUE" durable="true" auto-delete="false" exclusive="false" declared-by="connectAdmin" />
    <rabbit:queue name="FIRST_QUEUE" durable="true" auto-delete="false" exclusive="false" declared-by="connectAdmin" />

    <!-- 监听:消息到达队列时,通知消费者 -->
    <rabbit:listener-container connection-factory="connectionFactory">
        <rabbit:listener queues="FOURTH_QUEUE" ref="receiverFourth" />
    </rabbit:listener-container>

    <rabbit:listener-container connection-factory="connectionFactory">
        <rabbit:listener queues="THIRD_QUEUE" ref="receiverThird" />
    </rabbit:listener-container>

    <rabbit:listener-container connection-factory="connectionFactory">
        <rabbit:listener queues="MY_SECOND_QUEUE" ref="receiverSecond" />
    </rabbit:listener-container>

    <rabbit:listener-container connection-factory="connectionFactory">
        <rabbit:listener queues="FIRST_QUEUE" ref="messageReceiver" />
    </rabbit:listener-container>

    <!-- 发送消息,可通过connection-factory选择MQ节点 -->
    <rabbit:template id="amqpTemplate" connection-factory="connectionFactory"                 exchange="MY_DIRECT_EXCHANGE" />
</beans>
        ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml");
        MessageProducer messageProducer = (MessageProducer) context.getBean("messageProducer");
        int k = 100;
        while (k > 0) {
            messageProducer.sendMessage("第" + k + "次发送的消息");
            k--;
            try {
                Thread.sleep(1000);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

 org.springframework.amqp.core.AmqpTemplate负责发送消息:

@Service
public class Producer {
    private Logger logger = LoggerFactory.getLogger(MessageProducer.class);

    @Autowired
    @Qualifier("amqpTemplate")
    private AmqpTemplate amqpTemplate;

    @Autowired
    @Qualifier("amqpTemplate2")
    private AmqpTemplate amqpTemplate2;

    public void sendMessage(Object message) {
        amqpTemplate.convertAndSend("FirstKey", "[Direct,FirstKey] "+message);
        amqpTemplate.convertAndSend("SecondKey", "[Direct,SecondKey] "+message);
        amqpTemplate2.convertAndSend("msg.Third.send","[Topic,msg.Third.send] "+message);
        amqpTemplate2.convertAndSend("MY_FANOUT_EXCHANGE",null,"[Fanout] "+message);
    }
}

 org.springframework.amqp.core.MessageListener负责监听消息接收:

public class Consumer implements MessageListener {
    private Logger logger = LoggerFactory.getLogger(FirstConsumer.class);

    public void onMessage(Message message) {
        logger.info("received message : " + message.getBody());
    }
}

SpringBoot-RabbitMQ 

		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-amqp</artifactId>
		</dependency>

 spring-boot-starter-amqp默认支持RabbitMQ:

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter</artifactId>
      <version>2.1.5.RELEASE</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-messaging</artifactId>
      <version>5.1.7.RELEASE</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework.amqp</groupId>
      <artifactId>spring-rabbit</artifactId>
      <version>2.1.6.RELEASE</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <artifactId>http-client</artifactId>
          <groupId>com.rabbitmq</groupId>
        </exclusion>
      </exclusions>
    </dependency>
  </dependencies>

采用注解方式构造队列、交换机、绑定队列和交换机 

    @Bean("exchange1")
    public TopicExchange getTopicExchange(){
        return new TopicExchange("EXCHANGE1");
    }

    @Bean("queue1")
    public Queue getThirdQueue(){
        return new Queue("QUEUE1");
    }

    @Bean
    public Binding bindThird(@Qualifier("queue1") Queue queue, @Qualifier("exchange1") FanoutExchange exchange){
        return BindingBuilder.bind(queue).to(exchange);
    }

    @Bean
    public ConnectionFactory connectionFactory() throws Exception {
        CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory();
        cachingConnectionFactory.setUri("amqp://guest:guest@localhost:5672");
        return cachingConnectionFactory;
    }

@Bean
    public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMandatory(true);
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback(){
            public void returnedMessage(Message message,
                                        int replyCode,
                                        String replyText,
                                        String exchange,
                                        String routingKey){
                System.out.println("回发的消息:");
                System.out.println("replyCode: "+replyCode);
                System.out.println("replyText: "+replyText);
                System.out.println("exchange: "+exchange);
                System.out.println("routingKey: "+routingKey);
            }
        });

        rabbitTemplate.setChannelTransacted(true);

        rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
            public void confirm(CorrelationData correlationData, boolean ack, String cause) {
                if (!ack) {
                    System.out.println("发送消息失败:" + cause);
                    throw new RuntimeException("发送异常:" + cause);
                }
            }
        });



        return rabbitTemplate;
    }
AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext(BasicSender.class);
        RabbitTemplate rabbitTemplate = context.getBean(RabbitTemplate.class);
        rabbitTemplate.convertAndSend("","QUEUE1","-------- a direct msg");
@Component
@RabbitListener(queues = "QUEUE1")
public class FirstConsumer {

    @RabbitHandler
    public void process(String msg, Channel channel,long deliveryTag) throws IOException {
        channel.basicAck(deliveryTag, true);
        System.out.println(" first queue received msg : " + msg);
    }
}

 

DeadLetter(死信交换机):

消息超时未被消费者确认或者被消费者拒绝而且未设置重回队列,消息就会成为死信,进入死信交换机。

除了生产者可以设置消息过期时间,队列也可以统一设置进入队列的消息过期时间,此时最早的过期有效。

此外,队列达到最大长度,超过了Max length(最大消息数)或者Max length bytes(最大总消息字节数),最先入队的消息会进入死信交换机。

注:一个队列最多能放多少消息由Max length和Max length bytes决定。

超时未处理的消息可以从Original(普通)交换机对应的Original队列迁移到死信交换机。

应用举例:订单支付超时(Time To Line,超时):
订单支付消息在处理订单支付的队列中未被消费者消费,超时后经DeadLetter交换机路由到DeadLetter队列,被处理订单支付超时的消费者消费。
之所以不采用定时(如每分钟)查找数据库中支付超时的订单进行关闭,是因为订单量大,查询耗时,查到的支付超时的订单多时,处理也耗时。

超时未消费的消息处理:

        ConnectionFactory factory = new ConnectionFactory();
        factory.setUri("amqp://hao:123@127.0.0.1:5672");
        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();

        AMQP.BasicProperties properties = new AMQP.BasicProperties.Builder()
                .deliveryMode(2) // 持久化消息
                .contentEncoding("UTF-8")
                .expiration("10000") // 消息10秒不被消费,则过期,迁移到死信交换机
                .build();
        channel.basicPublish("", "USER_QUEUE", properties, "Hello RabbitMQ".getBytes());

        channel.close();
        conn.close();
        ConnectionFactory factory = new ConnectionFactory();
        factory.setUri("amqp://hao:123@127.0.0.1:5672");
        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();

        Map<String,Object> arguments = new HashMap<String,Object>();
        arguments.put("x-dead-letter-exchange","DEAD_LETTER_EXCHANGE");
        // arguments.put("x-expires",9000L); // 设置消息的过期时间
        // arguments.put("x-max-length", 4); // 如果设置了队列的最大长度,超过长度时,先入队的消息会被发送到死信交换机

        // 普通交换机对应的队列
        // String queue, boolean durable, boolean exclusive, boolean autoDelete, Map<String, Object> arguments
        channel.queueDeclare("USE_QUEUE", false, false, false, arguments);

        // 死信交换机
        channel.exchangeDeclare("GP_DEAD_LETTER_EXCHANGE","topic", false, false, false, null);
        // 死信交换机对应的队列
        channel.queueDeclare("GP_DEAD_LETTER_QUEUE", false, false, false, null);
        // 绑定,此处 Dead letter routing key 设置为 #
        channel.queueBind("DEAD_LETTER_QUEUE","GP_DEAD_LETTER_EXCHANGE","#");
        System.out.println(" Waiting for message....");

        // 消费者
        Consumer consumer = new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
                                       byte[] body) throws IOException {
                String msg = new String(body, "UTF-8");
                System.out.println("Received message : '" + msg + "'");
            }
        };

        // 监听消息到达死信队列,没有监听消息到达普通队列,也就是普通队列上的消息没有消费者消费
        // String queue, boolean autoAck, Consumer callback
        channel.basicConsume("DEAD_LETTER_QUEUE", true, consumer);

定时投递的消息处理:

x-delayed-message类型的交换机(需安装rabbitmq-delayed-message-exchange插件)。

解决订单超时未支付:也可以发送订单消息后,再延时发送一个消息,这个消息检查订单是否被支付。

        ConnectionFactory factory = new ConnectionFactory();
        factory.setUri("amqp://hao:123@127.0.0.1:5672");
        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();

        Date now = new Date();
        Calendar calendar = Calendar.getInstance();
        calendar.add(Calendar.SECOND, +10);//
        Date delayTime = calendar.getTime();
        SimpleDateFormat sf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
        String msg = "发送时间:" + sf.format(now) + ",投递时间:" + sf.format(delayTime);
        Map<String, Object> headers = new HashMap<String, Object>();
        // 延时时间
        headers.put("x-delay", delayTime.getTime() - now.getTime());

        AMQP.BasicProperties.Builder props = new AMQP.BasicProperties.Builder()
                .headers(headers);
        channel.basicPublish("DELAY_EXCHANGE", "DELAY_KEY", props.build(),
                msg.getBytes());

        channel.close();
        conn.close();
        ConnectionFactory factory = new ConnectionFactory();
        factory.setUri("amqp://guest:guest@127.0.0.1:5672");
        Connection conn = factory.newConnection();
        Channel channel = conn.createChannel();

        // 延时交换机
        Map<String, Object> argss = new HashMap<String, Object>();
        argss.put("x-delayed-type", "direct");
        channel.exchangeDeclare("DELAY_EXCHANGE", "x-delayed-message", false,
                false, argss);

        // 延时交换机对应的队列
        channel.queueDeclare("DELAY_QUEUE", false,false,false,null);

        // 绑定交换机与队列
        channel.queueBind("DELAY_QUEUE", "DELAY_EXCHANGE", "DELAY_KEY");

        // 创建消费者
        Consumer consumer = new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
                                       byte[] body) throws IOException {
                String msg = new String(body, "UTF-8");
                SimpleDateFormat sf=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
                System.out.println("收到消息:[" + msg + "]\n接收时间:" +sf.format(new Date()));
            }
        };

        // 获取消息
        // String queue, boolean autoAck, Consumer callback
        channel.basicConsume("DELAY_QUEUE", true, consumer);

交换机为“”,即默认交换机,此时routingKey是对应队列的名称:

channel.basicPublish("", "USE_QUEUE", properties, msg.getBytes());

持久化 

持久化可以提高RabbitMQ的可靠性,以防在RabbitMQ重启、关闭、宕机下的数据丢失。RabbitMQ可依靠Erlang自带的mnesia数据库实现持久化。https://www.jianshu.com/p/8f5868420991

1.交换器的持久化

交换器的持久化是在声明交换器的时候,将durable设置为true。如果交换器不设置持久化,那么在RabbitMQ交换器服务重启之后,相关的交换器信息会丢失,不过消息不会丢失,但是不能将消息发送到这个交换器。

2.队列对持久化

队列的持久化在声明队列的时候,将durable设置为true。如果队列不设置持久化,那么RabbitMQ交换器服务重启之后,相关的队列信息会丢失,同时队列中的消息也会丢失。

3.消息的持久化

消息的持久化是在BasicProperties中设置deliveryMode设置为2。队列的持久化能保证本身的元数据不会因为异常而丢失,但是不能保证内部所存在的消息不会丢失。要确保消息不丢失,需要将消息持久化。
如果将所有的消息都进行持久化操作,这样会严重影响RabbitMQ的性能。写入磁盘的速度比写入内存的速度慢很多。所以要在可靠性和吞吐量之间做权衡。

消费端限流

手动Ack下,消费者设置有几条消息未确认就停止收取消息。

channel.basicQos(int prefetchCount);

增加消费者消费效率

增加消费者或者消费者多线程消费。

动态更改

        ConnectionFactory connectionFactory = new CachingConnectionFactory(new URI("amqp://guest:guest@localhost:5672"));
        SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory = new SimpleRabbitListenerContainerFactory();
        simpleRabbitListenerContainerFactory.setConnectionFactory(connectionFactory);
        SimpleMessageListenerContainer container = simpleRabbitListenerContainerFactory.createListenerContainer();
        container.setConcurrentConsumers(1);
        container.setQueueNames("QUEUE1");
        container.setMessageListener(new MessageListener() {
            @Override
            public void onMessage(Message message) {
                System.out.println("收到消息:"+message);
            }
        });
        container.start();

消息、消息转换、动态更改队列等:

org.springframework.amqp.core.Message
org.springframework.amqp.support.converter.MessageConverter
org.springframework.amqp.rabbit.listener.MessageListenerContainer
 
  • 0
    点赞
  • 0
    收藏
    觉得还不错? 一键收藏
  • 打赏
    打赏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

风铃峰顶

你的鼓励将是我创作的最大动力

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值