Canal与RocketMQ的集成的思考与落地

前言

1. 在实现RocketMQ消费端时，想参考一些标准写法，搜github了解到
   rocketmq-externals项目，于是展开介绍一下

Message Connector 简介

2. 因为刚开始学习时用的是cannal的示例项目，想看看其他优秀框架是如何集成的，最终参考rocketmq-spring实现进行落地

消息路由（Message Connector）的几个概念：

1. 简单理解Message Connector就是借RocketMQ、kafka等从某个系统获取数据，借助消费消息写入到其他系统。主要由Source Connector，Sink Connector、Runtime组成。
   1.1 Source Connector负责从其它系统获取数据
   1.2 Sink Connector负责从Producer中消费消息，将数据写入到另外的系统
   1.3 Runtime是Source ，Sink connector的运行时环境，负责加载Connector，提供RESTful接口，启动Connector任务，集群节点之间服务发现，配置同步，消费进度保存，故障转移，负载均衡等能力
   

@RocketMQMessageListener注解的实现机制

因为在Canal集成RocketMq的消费示例是个demo演示，想参考优秀框架的集成机制转换成自己的知识库，所以有了下面的分析

入口ListenerContainerConfiguration#afterSingletonsInstantiated

1. ListenerContainerConfiguration加了@Configuration注解，并实现SmartInitializingSingleton的afterSingletonsInstantiated方法，所以Spring初始化会调用该方法
   
2. afterSingletonsInstantiated方法简要步骤
   2.1 扫描容器中所有带RocketMQMessageListener注解的Bean
   2.2 遍历这些bean，并逐一进行容器注册：registerContainer
3. registerContainer注册容器
   3.1 将容器bean注册到spring容器中，如果没有，则在获取Bean时会新建（懒加载）

   	GenericApplicationContext genericApplicationContext = (GenericApplicationContext) applicationContext;
   	genericApplicationContext.registerBean(containerBeanName, DefaultRocketMQListenerContainer.class,
           () -> createRocketMQListenerContainer(containerBeanName, bean, annotation));
       DefaultRocketMQListenerContainer container = genericApplicationContext.getBean(containerBeanName,
           DefaultRocketMQListenerContainer.class);
   

   3.2 启动容器：DefaultRocketMQListenerContainer#start

消费端订阅主题

1. 因为DefaultRocketMQListenerContainer实现了InitializingBean接口的afterPropertiesSet方法，所以在getBean时会触发
   
2. 在初始化RocketMQ消费端时，关键就是订阅主题和设置监听器

   	private void initRocketMQPushConsumer() throws MQClientException {
   		.......
   		consumer.subscribe(topic, selectorExpression); //订阅主题
   		........
   		consumer.setMessageListener(new DefaultMessageListenerConcurrently());//设置监听器
   		
   	}
   

消费端监听消息

1. 在从broker里拉取到消息后，进行回调成功处理：DefaultMQPushConsumerImpl$PullCallback#onSuccess

   	// 将拉取到的消息放入processQueue队列中
   	boolean dispatchToConsume = processQueue.putMessage(pullResult.getMsgFoundList());                     
   	// 将要处理的消息封装成消费请求丢到线程池中
   	DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(
                                   pullResult.getMsgFoundList(),
                                   processQueue,
                                   pullRequest.getMessageQueue(),
                                   dispatchToConsume);
   

   ConsumeMessageConcurrentlyService#submitConsumeRequest 并发消费
   public void submitConsumeRequest(xxx,xxx,xxx){
   	 ConsumeRequest consumeRequest = new ConsumeRequest(msgs, processQueue, messageQueue);
   	 this.consumeExecutor.submit(consumeRequest);
   }
   

2. 消息处理：ConsumeMessageConcurrentlyService$ConsumeRequest#run

   MessageListenerConcurrently listener = 	ConsumeMessageConcurrentlyService.this.messageListener;
   // 回调用户的监听器
   status = listener.consumeMessage(Collections.unmodifiableList(msgs), context);
   

最后方案

1. 扫描Spring容器中@RowDataChangeListener的类，设置到从cannal客户端取出数据的处理
   1.1 定义RowDataListener接口

   	public interface RowDataListener {
   	    void onMessage(List<CanalEntry.Entry> entries);
   	}
   

   1.2 实现RowDataListener接口，并添加@RowDataChangeListener注解

   	@RowDataChangeListener
   	@Service
   	public class RowDataChangeService implements RowDataListener {
   			@Override
   		    public void onMessage(List<CanalEntry.Entry> entries) {
   		        printEntry(entries);
   		    }
   	}
   

   1.3 定义Config，初始化时添加@RowDataChangeListener作为数据处理

   	@Configuration
   	@EnableConfigurationProperties(RocketMQProperties.class)
   	public class RocketMqConfig implements ApplicationContextAware,SmartInitializingSingleton {
   		@Override
   	    public void afterSingletonsInstantiated() {
   	        Map<String, Object> beans =  this.applicationContext.getBeansWithAnnotation(RowDataChangeListener.class);
   	        if(beans.size() > 1){
   	            throw new IllegalStateException("@RowDataChangeListener只能存在一个");
   	        }
   	        beans.forEach(this::start);
   	    }
   		
   		private void start(String beanName, Object bean){
   	        if(bean instanceof RowDataListener){
   	            RocketMQCanalConnector connector = createRocketMQCanalConnector();
   	            process(connector, (RowDataListener) bean);
   	        }
   	    }
   		// 简写部分
   		private void process(RocketMQCanalConnector connector, RowDataListener rowDataListener){
   			 connector.connect();
               connector.subscribe();
               List<Message> messages = connector.getListWithoutAck(1000L, TimeUnit.MILLISECONDS); 
               for (Message message : messages) {
               	rowDataListener.onMessage(message.getEntries());
               }
                connector.ack(); // 提交确认
   		}
   	}