RocketMQ普通消费、广播消费、顺序消费、延时消费、过滤消费、事务消息

RocketMQ

一、安装RocketMQ并启动Name Server（sh bin/mqnamesrv）启动 Broker（sh bin/mqbroker -n localhost:9876）

二、依赖（用于构建基于消息的微服务应用框架）

①、Binder: 跟外部消息中间件集成的组件，用来创建 Binding，各消息中间件都有自己的 Binder 实现。
Kafka 的实现 KafkaMessageChannelBinder，RabbitMQ 的实现 RabbitMessageChannelBinder 以及 RocketMQ 的实现 RocketMQMessageChannelBinder

②、Binding: 包括 Input Binding 和 Output Binding。在消息中间件与应用程序提供的 Provider 和 Consumer 之间提供了一个桥梁，实现了开发者只需使用应用程序的 Provider 或 Consumer 生产或消费数据即可，屏蔽了开发者与底层消息中间件的接触

<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rocketmq</artifactId>
</dependency>

三、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t test-topic

四、配置input和output信息，并配合@EnableBinding注解使其生效

spring.application.name=rocketmq-example
server.port=28081

@SpringBootApplication
@EnableBinding({Source.class,Sink.class})
public class RocketMQApplication{
	
	public static void main(String[] args) {
		SpringApplication.run(RocketMQApplication.class, args);
	}
}

配置Binding信息

#配置rocketmq的nameserver地址
spring.cloud.stream.rocketmq.binder.name-server=127.0.0.1:9876

# 定义name为output的binding
spring.cloud.stream.bindings.output.destination=test-topic
spring.cloud.stream.bindings.output.content-type=application/json

# 定义name为input的binding
spring.cloud.stream.bindings.input.destination=test-topic
spring.cloud.stream.bindings.input.content-type=application/json
spring.cloud.stream.bindings.input.group=test-group

启动应用：

• IDE 直接启动：找到主类 RocketMQApplication，执行 main 方法启动应用
• 打包编译后启动：首先执行 mvn clean package 将工程编译打包，然后执行 java -jar rocketmq-example.jar 启动应用

五、消息处理

• 使用name为output对应的binding发送消息到test-topic这个topic
• 使用2个input binding订阅数据：input1: 订阅 topic 为 test-topic 的消息，顺序消费所有消息(顺序消费的前提是所有消息都在一个 MessageQueue 中)；input2: 订阅 topic 为 test-topic 的消息，异步消费 tags 为 tagStr 的消息，Consumer 端线程池个数为20

配置信息

spring.cloud.stream.rocketmq.binder.name-server=127.0.0.1:9876

spring.cloud.stream.bindings.output.destination=test-topic
spring.cloud.stream.bindings.output.content-type=application/json

spring.cloud.stream.bindings.input1.destination=test-topic
spring.cloud.stream.bindings.input1.content-type=text/plain
spring.cloud.stream.bindings.input1.group=test-group1
spring.cloud.stream.rocketmq.bindings.input1.consumer.orderly=true

spring.cloud.stream.bindings.input2.destination=test-topic
spring.cloud.stream.bindings.input2.content-type=text/plain
spring.cloud.stream.bindings.input2.group=test-group2
spring.cloud.stream.rocketmq.bindings.input2.consumer.orderly=false
spring.cloud.stream.rocketmq.bindings.input2.consumer.tags=tagStr
spring.cloud.stream.bindings.input2.consumer.concurrency=20

使用MessageChannel进行消息发送

public class ProducerRunner implements CommandLineRunner{
	
	@Autowired
	private MessageChannel output;//获取name为output的binding

	@Override
	public void run(String...args)throws Exception{
		Map<String,Object> headers = new HashMap<>();
		headers.put(MessageConst.PROPERTY_TAGS,"tagStr");
		Message message = MessageBuilder.createMessage(msg, new MessageHeaders(headers));
        output.send(message);
	}
}

或者使用RocketMQ原生API进行消息发送

public class RocketMQProducer{
	
	DefaultMQProducer producer = new DefaultMQProducer("producer_group");
	producer.setNamesrvAddr("127.0.0.1:9876");
    producer.start();

	Message msg = new Message("test-topic", "tagStr", "message from rocketmq producer".getBytes());
    producer.send(msg);
}

使用@StreamListener注解接收消息

@Service
public class ReceiveService{
	
	@StreamListener("input1")
	public void receiveInput1(String receiveMsg) {
		System.out.println("input1 receive: " + receiveMsg);
	}

	@StreamListener("input2")
	public void receiveInput2(String receiveMsg) {
		System.out.println("input2 receive: " + receiveMsg);
	}
}

广播消费

广播会发送消息给所有消费者。如果你想同一消费组下所有消费者接收到同一个topic下的消息

一、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t broadcast

二、生产者

server:
	port: 28085
spring:
	application:
		name: rocketmq-broadcast-producer-example
	cloud:
		stream:
			rocketmq:
				binder:
					name-server: localhost:9876
				bingdings:
					producer-out-0:
						producer:
							group: output_1
			bindings:
				producer-out-0:
					destination:broadcast
logging:
	level：
		org.springframework.context.support: debug						

使用ApplicationRunner和StreamBridge发送消息

@SpringBootApplication
public class RocketMQBrodcastProducerApplication{
	
	private static final Logger log = LoggerFactory
         .getLogger(RocketMQBroadcastProducerApplication.class);
	
	@Autowired
	private StreamBridge streamBridge;

	public static void main(String[] args){
		SpringApplication.run(RocketMQBroadcastProducerApplication.class, args);
	}

	@Bean
	public ApplicationRunner producer(){
		return args - >{
			for (int i = 0; i < 100; i++) {
	            String key = "KEY" + i;
	            Map<String, Object> headers = new HashMap<>();
	            headers.put(MessageConst.PROPERTY_KEYS, key);
	            headers.put(MessageConst.PROPERTY_ORIGIN_MESSAGE_ID, i);
	            Message<SimpleMsg> msg = new GenericMessage<SimpleMsg>(new SimpleMsg("Hello RocketMQ " + i), headers);
	            streamBridge.send("producer-out-0", msg);
         }	
		};
	}
}

启动两个消费者实例

消费者1

server:
  port: 28084
spring:
  application:
    name: rocketmq-broadcast-consumer1-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          consumer-in-0:
            consumer:
              messageModel: BROADCASTING
      bindings:
        consumer-in-0:
          destination: broadcast
          group: broadcast-consumer
logging:
  level:
    org.springframework.context.support: debug

@SpringBootApplication
public class RocketMQBroadcastConsumer1Application {
   private static final Logger log = LoggerFactory
         .getLogger(RocketMQBroadcastConsumer1Application.class);

   public static void main(String[] args) {
      SpringApplication.run(RocketMQBroadcastConsumer1Application.class, args);
   }
	
	@Bean
	public Consumer<Message<SimpleMsg>> consumer(){
		return msg -> {
			log.info(Thread.currentThread().getName() + " Consumer1 Receive New Messages: " + msg.getPayload().getMsg());
		};
	}
}

消费者2

server:
  port: 28083
spring:
  application:
    name: rocketmq-broadcast-consumer2-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          consumer-in-0:
            consumer:
              messageModel: BROADCASTING
      bindings:
        consumer-in-0:
          destination: broadcast
          group: broadcast-consumer
logging:
  level:
    org.springframework.context.support: debug

@SpringBootApplication
public class RocketMQBroadcastConsumer2Application {
   private static final Logger log = LoggerFactory
         .getLogger(RocketMQBroadcastConsumer2Application.class);

   public static void main(String[] args) {
      SpringApplication.run(RocketMQBroadcastConsumer2Application.class, args);
   }

   @Bean
   public Consumer<Message<SimpleMsg>> consumer() {
      return msg -> {
         log.info(Thread.currentThread().getName() + " Consumer2 Receive New Messages: " + msg.getPayload().getMsg());
      };
   }
}

顺序消费

• 全局顺序：对于指定的一个Topic，所有消息按照严格的先入先出FIFO（First In First Out）的顺序进行发布和消费。
• 分区顺序：对于指定的一个Topic，所有消息根据Sharding Key进行区块分区。同一个分区内的消息按照严格的FIFO顺序进行发布和消费。Sharding Key是顺序消息中用来区分不同分区的关键字段，和普通消息的Key是完全不同的概念。

一、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t orderly

二、application.yml

server:
  port: 28082
spring:
  application:
    name: rocketmq-orderly-consume-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          producer-out-0:
            producer:
              group: output_1
              # 定义messageSelector
              messageQueueSelector: orderlyMessageQueueSelector
          consumer-in-0:
            consumer:
              # tag: {@code tag1||tag2||tag3 }; sql: {@code 'color'='blue' AND 'price'>100 } .
              subscription: 'TagA || TagC || TagD'
              push:
                orderly: true
      bindings:
        producer-out-0:
          destination: orderly
        consumer-in-0:
          destination: orderly
          group: orderly-consumer

logging:
  level:
    org.springframework.context.support: debug

三、MessageQueueSelector

选择适合自己的分区选择算法，保证同一个参数得到的结果相同。

@Component
public class OrderlyMessageQueueSelector implements MessageQueueSelector{
	
	private static final Logger log = LoggerFactory
         .getLogger(OrderlyMessageQueueSelector.class);

	@Override
	public MessageQueue select(List<MessageQueue> mqs,Message msg,Object arg){
		Integer id = (Integer) ((MessageHeaders) arg).get(MessageConst.PROPERTY_ORIGIN_MESSAGE_ID);
        String tag = (String) ((MessageHeaders) arg).get(MessageConst.PROPERTY_TAGS);
        int index = id % RocketMQOrderlyConsumeApplication.tags.length % mqs.size();
        return mqs.get(index);
	}
}

四、生产者和消费者

@SpringBootApplication
public class RocketMQOrderlyConsumeApplication {
   private static final Logger log = LoggerFactory
         .getLogger(RocketMQOrderlyConsumeApplication.class);
	
	@Autowired
	private StreamBridge streamBridge;
	
	public static final String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};

	public static void main(String[] args) {
      SpringApplication.run(RocketMQOrderlyConsumeApplication.class, args);
    }
	
	//消息生产者
	@Bean
	public ApplicationRunner producer(){
		return args -> {
			for (int i = 0; i < 100; i++) {
	            String key = "KEY" + i;
	            Map<String, Object> headers = new HashMap<>();
	            headers.put(MessageConst.PROPERTY_KEYS, key);
	            headers.put(MessageConst.PROPERTY_TAGS, tags[i % tags.length]);
	            headers.put(MessageConst.PROPERTY_ORIGIN_MESSAGE_ID, i);
	            Message<SimpleMsg> msg = new GenericMessage(new SimpleMsg("Hello RocketMQ " + i), headers);
	            streamBridge.send("producer-out-0", msg);
         	}
		};
	}

	//消息消费者
	@Bean
   public Consumer<Message<SimpleMsg>> consumer() {
      return msg -> {
         String tagHeaderKey = RocketMQMessageConverterSupport.toRocketHeaderKey(
               MessageConst.PROPERTY_TAGS).toString();
         log.info(Thread.currentThread().getName() + " Receive New Messages: " + msg.getPayload().getMsg() + " TAG:" +
               msg.getHeaders().get(tagHeaderKey).toString());
         try {
            Thread.sleep(100);
         }
         catch (InterruptedException ignored) {
         }
      };
   }
	
}

延时消费

Producer将消息发送到消息队列RocketMQ服务端，但并不期望立马投递这条消息，而是延迟一定时间后才投递到Consumer进行消费，该消息即延时消息。

一、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t delay

二、application.yml

server:
  port: 28086
spring:
  application:
    name: rocketmq-delay-consume-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          producer-out-0:
            producer:
              group: output_1
      bindings:
        producer-out-0:
          destination: delay
        consumer-in-0:
          destination: delay
          group: delay-group
logging:
  level:
    org.springframework.context.support: debug

三、生产者与消费者

@SpringBootApplication
public class RocketMQDelayConsumeApplication {
   private static final Logger log = LoggerFactory
         .getLogger(RocketMQDelayConsumeApplication.class);
   @Autowired
   private StreamBridge streamBridge;

   public static void main(String[] args) {
      SpringApplication.run(RocketMQDelayConsumeApplication.class, args);
   }

   @Bean
   public ApplicationRunner producerDelay() {
      return args -> {
         for (int i = 0; i < 100; i++) {
            String key = "KEY" + i;
            Map<String, Object> headers = new HashMap<>();
            headers.put(MessageConst.PROPERTY_KEYS, key);
            headers.put(MessageConst.PROPERTY_ORIGIN_MESSAGE_ID, i);
   			// 设置延时等级1~10
            headers.put(MessageConst.PROPERTY_DELAY_TIME_LEVEL, 2);
            Message<SimpleMsg> msg = new GenericMessage(new SimpleMsg("Delay RocketMQ " + i), headers);
            streamBridge.send("producer-out-0", msg);
         }
      };
   }

   @Bean
   public Consumer<Message<SimpleMsg>> consumer() {
      return msg -> {
         log.info(Thread.currentThread().getName() + " Consumer Receive New Messages: " + msg.getPayload().getMsg());
      };
   }
}

过滤消息

支持tag过滤或者sql过滤，设置spring.cloud.stream.rocketmq.bindings..consumer.subscription即可。

tag示例: tag:red || blue

sql示例: sql:(color in (‘red1’, ‘red2’, ‘red4’) and price>3)

一、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t sql

二、配置文件

server:
  port: 28087
spring:
  application:
    name: rocketmq-sql-consume-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          producer-out-0:
            producer:
              group: output_1
          consumer-in-0:
            consumer:
              # tag: {@code tag1||tag2||tag3 }; sql: {@code 'color'='blue' AND 'price'>100 } .
              subscription: sql:(color in ('red1', 'red2', 'red4') and price>3)
      bindings:
        producer-out-0:
          destination: sql
        consumer-in-0:
          destination: sql
          group: sql-group
logging:
  level:
    org.springframework.context.support: debug

三、代码

@SpringBootApplication
public class RocketMQSqlConsumeApplication {
   private static final Logger log = LoggerFactory
         .getLogger(RocketMQSqlConsumeApplication.class);
   @Autowired
   private StreamBridge streamBridge;
   public static void main(String[] args) {
      SpringApplication.run(RocketMQSqlConsumeApplication.class, args);
   }

   /**
    * color array.
    */
   public static final String[] color = new String[] {"red1", "red2", "red3", "red4", "red5"};

   /**
    * price array.
    */
   public static final Integer[] price = new Integer[] {1, 2, 3, 4, 5};

   @Bean
   public ApplicationRunner producer() {
      return args -> {
         for (int i = 0; i < 100; i++) {
            String key = "KEY" + i;
            Map<String, Object> headers = new HashMap<>();
            headers.put(MessageConst.PROPERTY_KEYS, key);
            headers.put("color", color[i % color.length]);
            headers.put("price", price[i % price.length]);
            headers.put(MessageConst.PROPERTY_ORIGIN_MESSAGE_ID, i);
            Message<SimpleMsg> msg = new GenericMessage(new SimpleMsg("Hello RocketMQ " + i), headers);
            streamBridge.send("producer-out-0", msg);
         }
      };
   }

   @Bean
   public Consumer<Message<SimpleMsg>> consumer() {
      return msg -> {
         String colorHeaderKey = "color";
         String priceHeaderKey = "price";
         log.info(Thread.currentThread().getName() + " Receive New Messages: " + msg.getPayload().getMsg() + " COLOR:" +
               msg.getHeaders().get(colorHeaderKey).toString() + " " +
               "PRICE: " + msg.getHeaders().get(priceHeaderKey).toString());
      };
   }
}

常见问题：

MQClientException: The broker does not support consumer to filter message by SQL92

1. 修改 RocketMQ 服务端配置文件。 在 conf/2m-2s-async/broker-a.properties 配置文件末尾添加 enablePropertyFilter=true
2. 重启 mqbroker 并指定配置文件。 mqbroker 启动时指定配置文件：conf/2m-2s-async/broker-a.properties，例如：
   bin/mqbroker -n 127.0.0.1:9876 -c conf/2m-2s-async/broker-a.properties autoCreateTopicEnable=true

事务消息

• TransactionStatus.CommitTransaction: 提交事务，意味着消费者可以消费事务
• TransactionStatus.RollbackTransaction: 回滚事务，消息将被删除，并且不允许被消费
• TransactionStatus.Unknown: 中间状态，意味着MQ需要回查最终状态

一、创建Topic

sh bin/mqadmin updateTopic -n localhost:9876 -c DefaultCluster -t tx

二、配置文件

server:
  port: 28088
spring:
  application:
    name: rocketmq-tx-example
  cloud:
    stream:
      function:
        definition: consumer;
      rocketmq:
        binder:
          name-server: localhost:9876
        bindings:
          producer-out-0:
            producer:
              group: output_1
              transactionListener: myTransactionListener
              producerType: Trans
      bindings:
        producer-out-0:
          destination: tx
        consumer-in-0:
          destination: tx
          group: tx-group
logging:
  level:
    org.springframework.context.support: debug

三、TransactionListenerImpl执行本地事务

@Component("myTransactionListener")
public class TransactionListenerImpl implements TransactionListener {
	
	@Override
	public LocalTransactionState executeLocalTransaction(Message msg, Object arg){
		Object num = msg.getProperty("test");
		if("1".equals(num)){
			System.out.println();
			return LocalTransationState.UNKNOW;
		}else if("2".equals(num)){
			System.out.println();
			return LocalTransactionState.ROLLBACK_MESSAGE;
		}
		System.out.println("executer: " + new String(msg.getBody()) + " commit");
		return LocalTransactionState.COMMIT_MESSAGE;
	}

	@Override
	public LocalTransactionState checkLocalTransaction(MessageExt msg){
		System.out.println("check: " + new String(msg.getBody()));
		return LocalTransactionState.COMMIT_MESSAGE;
	}
}

四、生产者和消费者

@SpringBootApplication
public class RocketMQTxApplication{
	
	private static final Logger log = LoggerFactory.getLogger(RocketMQTxApplication.class);

	@Autowired
	private StreamBridge streamBridge;

	public static void main(String[] args){
		SpringApplication.run(RocketMQTxApplication.class, args);
	}

	@Bean
	public ApplicationRunner producer(){
		return args -> {
			for(int i = 1; i <= 4; i++){
				MessageBuilder builder = MessageBuilder.withPayLoad(new SimpleMsg("Hello Tx msg " + i));
				builder.setHeader("test", String.valueOf(i))
						.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON);
				builder.setHeader(RocketMQConst.USER_TRANSACTIONAL_ARGS, "binder");
				Message<SimpleMsg> msg = builder.build();
				streamBridge.send("producer-out-0", msg);
				System.out.println("send Msg:" + msg.toString());
			}
		}
	}

	@Bean
	public Consumer<Message<SimpleMsg>> consumer(){
		return msg -> {
			Object arg = msg.getHeaders();
			log.info(Thread.currentThread().getName() + " Receive New Messages: " + msg.getPayload().getMsg() + " ARG:"
				+ arg.toString());
		};
	}
}