RocketMq源码解读（二）

本章主要介绍的内容有：消息过滤服务filterSrv

一、什么是filterSrv

1、filtersrv

filtersrv是RocketMQ中的消息过滤服务，是介于consumer和broker之间的代理。主要作用是在consumer端定义好filter过滤规则，动态编译成class，根据定义的规则，从broker拉取消息，然后将拉取到的满足过滤条件的消息返回给consumer。所以整体的工作流程有：

2、为什么要单独建立filtersrv服务

• 在broker端增加消息的过滤：优点是减少对于consumer的无效网络传输，缺点是增加了broker端的复杂度，并且当规则多样时，对broker的负担较大。
• 在consumer端增加消息的过滤：优点是实现灵活，可以自主的定义灵活规则，缺点是有很多无用的消息传输到consumer，耗费网路带宽和消息的消费速度。

在这种情况下，出现了filtersrv服务，它的思想和nginx类似，做一层简单的代理，这样减少了broker的负担，也减少了consumer接收到的无用消息。不过，这也带来了filtersrv的网络开销。

 二、filterSrv的源码解读

1、先看一段消费端的简单代码

public void subscribe(String topic, String fullClassName, String filterClassSource) throws MQClientException {
    this.defaultMQPushConsumerImpl.subscribe(topic, fullClassName, filterClassSource);
}

在其实现方法中，会向所有的broker发送心跳
   if (this.mQClientFactory != null) {
           this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
    }

发送心跳的时候，会自动上传过滤类到filtersrv， 

this.uploadFilterClassToAllFilterServer(consumerGroup, className, topic, filterClassSource);

所以消费端在消费的时候，需要通过上传的过滤函数过滤。

2、Broker和filtersrv的绑定关系

filtersrv的配置信息放在flitersrvConfig类中，

public class FiltersrvConfig {
    // 通过环境变量确定home dir
    private String rocketmqHome = System.getProperty(MixAll.ROCKETMQ_HOME_PROPERTY,
        System.getenv(MixAll.ROCKETMQ_HOME_ENV));

    // 获取名字空间的地址
    @ImportantField
    private String namesrvAddr = System.getProperty(MixAll.NAMESRV_ADDR_PROPERTY,
        System.getenv(MixAll.NAMESRV_ADDR_ENV));
    // 和哪一个broker连接
    private String connectWhichBroker = "127.0.0.1:10911";
    // Filter Server对外服务的IP
    private String filterServerIP = RemotingUtil.getLocalAddress();
    // 消息超过指定大小，开始压缩
    private int compressMsgBodyOverHowmuch = 1024 * 8;
    private int zipCompressLevel = 5;
     // 是否允许client上传过滤类
    private boolean clientUploadFilterClassEnable = true;
    // 过滤类的仓库地址
    private String filterClassRepertoryUrl = "http://fsrep.tbsite.net/filterclass";
    private int fsServerAsyncSemaphoreValue = 2048;
    private int fsServerCallbackExecutorThreads = 64;
    private int fsServerWorkerThreads = 64;
    ...
}

这其中有两个很关键的参数：其中namesrvAddr和broker一样，而connectWhichBroker表示当前的filtersrv是属于哪一个broker。

filter服务在启动的时候，向broker注册，具体的实现方法在在FiltersrvController的initialize方法的时候调用了FilterServerOuterAPI的registerFilterServerToBroker方法将自己注册到broker里面去了。并且间隔的周期是10秒中。

  public boolean initialize() {
        // 打印配置信息
        MixAll.printObjectProperties(log, this.filtersrvConfig);
        // 启动远程的netty服务
        this.remotingServer = new NettyRemotingServer(this.nettyServerConfig);
        // 创建消费线程池
        this.remotingExecutor =
            Executors.newFixedThreadPool(nettyServerConfig.getServerWorkerThreads(),
                new ThreadFactoryImpl("RemotingExecutorThread_"));
        // 注册处理器
        this.registerProcessor();
        // 固定间隔注册到Broker
        this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
            @Override
            public void run() {
                FiltersrvController.this.registerFilterServerToBroker();
            }
        }, 15, 10, TimeUnit.SECONDS);
        // 设置最长阻塞时间
        this.defaultMQPullConsumer.setBrokerSuspendMaxTimeMillis(this.defaultMQPullConsumer
            .getBrokerSuspendMaxTimeMillis() - 1000);
        this.defaultMQPullConsumer.setConsumerTimeoutMillisWhenSuspend(this.defaultMQPullConsumer
            .getConsumerTimeoutMillisWhenSuspend() - 1000);
        // 配置名字空间地址
        this.defaultMQPullConsumer.setNamesrvAddr(this.filtersrvConfig.getNamesrvAddr());
        this.defaultMQPullConsumer.setInstanceName(String.valueOf(UtilAll.getPid()));

        return true;
    }

在FilterServerOuterAPI中，会向broker服务发送方注册请求，

public RegisterFilterServerResponseHeader registerFilterServerToBroker(
        final String brokerAddr, final String filterServerAddr
    ) throws RemotingCommandException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException, InterruptedException, MQBrokerException {
        RegisterFilterServerRequestHeader requestHeader = new RegisterFilterServerRequestHeader();
        requestHeader.setFilterServerAddr(filterServerAddr);
        // 发送注册filter的命令
        RemotingCommand request =
            RemotingCommand.createRequestCommand(RequestCode.REGISTER_FILTER_SERVER, requestHeader);

        RemotingCommand response = this.remotingClient.invokeSync(brokerAddr, request, 3000);
        assert response != null;
        switch (response.getCode()) {
            case ResponseCode.SUCCESS: {
                RegisterFilterServerResponseHeader responseHeader =
                    (RegisterFilterServerResponseHeader) response
                        .decodeCommandCustomHeader(RegisterFilterServerResponseHeader.class);

                return responseHeader;
            }
            default:
                break;
        }
        // 如果注册不成功，会抛出异常
        throw new MQBrokerException(response.getCode(), response.getRemark());
    }

在broker端有接收处理filter注册的处理，

// 在adminBrokerProcessor里，有处理netty请求
case RequestCode.REGISTER_FILTER_SERVER:
    return this.registerFilterServer(ctx, request);

// 可以看到，一个broker可以有多个filtersrv服务，封装到filterServerTable里
 public void registerFilterServer(final Channel channel, final String filterServerAddr) {
        FilterServerInfo filterServerInfo = this.filterServerTable.get(channel);
        if (filterServerInfo != null) {
            filterServerInfo.setLastUpdateTimestamp(System.currentTimeMillis());
        } else {
            filterServerInfo = new FilterServerInfo();
            filterServerInfo.setFilterServerAddr(filterServerAddr);
            filterServerInfo.setLastUpdateTimestamp(System.currentTimeMillis());
            this.filterServerTable.put(channel, filterServerInfo);
            log.info("Receive a New Filter Server<{}>", filterServerAddr);
        }
    }

在broker向namesrv注册自己的时候，会带着filtersrv的地址，

public List<String> buildNewFilterServerList() {
    List<String> addr = new ArrayList<String>();
    Iterator<Entry<Channel, FilterServerInfo>> it = this.filterServerTable.entrySet().iterator();
    while (it.hasNext()) {
        Entry<Channel, FilterServerInfo> next = it.next();
        addr.add(next.getValue().getFilterServerAddr());
    }
    return addr;
}

所以在namesrv里面，会有处理registerBrokerWithFilterServer处理方法，将broker和filterServerList放到filterserverTable的hashmap中。

public RegisterBrokerResult registerBroker(
        final String clusterName,final String brokerAddr,
        final String brokerName, final long brokerId,
        final String haServerAddr, final TopicConfigSerializeWrapper topicConfigWrapper,
        final List<String> filterServerList, final Channel channel){
    ...
     if (filterServerList != null) {
         if (filterServerList.isEmpty()) {
              this.filterServerTable.remove(brokerAddr);
         } else {
              this.filterServerTable.put(brokerAddr, filterServerList);
         }
     }
}

namesrv中通过topic查找对应的broker时，会将信息封装到TopicRouteData返回给client，

public TopicRouteData pickupTopicRouteData(final String topic){
    opicRouteData topicRouteData = new TopicRouteData();
    boolean foundQueueData = false;
    boolean foundBrokerData = false;
    Set<String> brokerNameSet = new HashSet<String>();
    List<BrokerData> brokerDataList = new LinkedList<BrokerData>();
    topicRouteData.setBrokerDatas(brokerDataList);
    HashMap<String, List<String>> filterServerMap = new HashMap<String, List<String>>();
    // 返回数据中带有过滤map
    topicRouteData.setFilterServerTable(filterServerMap);
    ... 
}

3、 filtersrv如何处理来自consumer的MessageFilter

当consumer发送注册filtersrv的心跳包时，filtersrv接收到请求根据不同的code处理响应。请求code分为RequestCode.REGISTER_MESSAGE_FILTER_CLASS和RequestCode.PULL_MESSAGE。处理注册MessageFilter实现类是方法registerMessageFilterClass，最终会在FilterClassManager动态编译来自consumer的java源码生成一个MessageFilter的类对象，然后反射生成对象。具体逻辑如下：

private RemotingCommand registerMessageFilterClass(ChannelHandlerContext ctx, RemotingCommand request) throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(null);
    final RegisterMessageFilterClassRequestHeader requestHeader =
            (RegisterMessageFilterClassRequestHeader) request.decodeCommandCustomHeader(RegisterMessageFilterClassRequestHeader.class);

    try {
        boolean ok = this.filtersrvController.getFilterClassManager().registerFilterClass(requestHeader.getConsumerGroup(),requestHeader.getTopic(),requestHeader.getClassName(),requestHeader.getClassCRC(),request.getBody());
        if (!ok) {
            throw new Exception("registerFilterClass error");
        }
     } catch (Exception e) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark(RemotingHelper.exceptionSimpleDesc(e));
        return response;
     }
     response.setCode(ResponseCode.SUCCESS);
     response.setRemark(null);
     return response;
}

注册过滤类实现主要是建立一个消费组和topic确定的class map。

public boolean registerFilterClass(final String consumerGroup, final String topic,
        final String className, final int classCRC, final byte[] filterSourceBinary) {
    final String key = buildKey(consumerGroup, topic);
    // 判断是否要注册新的过滤类
    boolean registerNew = false;
    FilterClassInfo filterClassInfoPrev = this.filterClassTable.get(key);
    if (null == filterClassInfoPrev) {
        registerNew = true;
    } else {
        if (this.filtersrvController.getFiltersrvConfig().isClientUploadFilterClassEnable()) {
            if (filterClassInfoPrev.getClassCRC() != classCRC && classCRC != 0) { 
                registerNew = true;
            }
        }
    }
    // 注册新的过滤类
    if (registerNew) {
         synchronized (this.compileLock) {
            ...
            // 放到内存缓存中
            this.filterClassTable.put(key, filterClassInfoNew);
         }
    }
    return true;
}    

在consumer pull消息的时候，会有

private RemotingCommand pullMessageForward(final ChannelHandlerContext ctx, final RemotingCommand request) throws Exception {
    // 构造返回响应
    final RemotingCommand response = RemotingCommand.createResponseCommand(PullMessageResponseHeader.class);
    final PullMessageResponseHeader responseHeader = (PullMessageResponseHeader) response.readCustomHeader();
    final PullMessageRequestHeader requestHeader =
            (PullMessageRequestHeader) request.decodeCommandCustomHeader(PullMessageRequestHeader.class);
    // 构造上下文
    final FilterContext filterContext = new FilterContext();
    filterContext.setConsumerGroup(requestHeader.getConsumerGroup());
    response.setOpaque(request.getOpaque());
    DefaultMQPullConsumer pullConsumer = this.filtersrvController.getDefaultMQPullConsumer();
    // 校验Topic过滤类是否完整，如果没有过滤类，返回错误码
    final FilterClassInfo findFilterClass = this.filtersrvController.getFilterClassManager().findFilterClass(requestHeader.getConsumerGroup(), requestHeader.getTopic());
    if (null == findFilterClass) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("Find Filter class failed, not registered");
        return response;
    }
    if (null == findFilterClass.getMessageFilter()) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
        response.setRemark("Find Filter class failed, registered but no class");
        return response;
    }

    // 设置下次请求从 Broker主节点。
    responseHeader.setSuggestWhichBrokerId(MixAll.MASTER_ID);
    // 构造从broker拉取小气的参数
    MessageQueue mq = new MessageQueue();
    mq.setTopic(requestHeader.getTopic());
    mq.setQueueId(requestHeader.getQueueId());
    mq.setBrokerName(this.filtersrvController.getBrokerName());
    long offset = requestHeader.getQueueOffset();
    int maxNums = requestHeader.getMaxMsgNums();

    final PullCallback pullCallback = new PullCallback() {
        @Override
        public void onSuccess(PullResult pullResult) {
        responseHeader.setMaxOffset(pullResult.getMaxOffset());
        responseHeader.setMinOffset(pullResult.getMinOffset());
        responseHeader.setNextBeginOffset(pullResult.getNextBeginOffset());
        response.setRemark(null);
        switch (pullResult.getPullStatus()) {
            case FOUND:
                response.setCode(ResponseCode.SUCCESS);
                List<MessageExt> msgListOK = new ArrayList<MessageExt>();
                try {
                    for (MessageExt msg : pullResult.getMsgFoundList()) {
                        // 使用过滤类过滤消息
                        boolean match = findFilterClass.getMessageFilter().match(msg, filterContext);
                        if (match) {
                             msgListOK.add(msg);
                         }
                     }
                     // 消息不为空，返回构造消息
                     if (!msgListOK.isEmpty()) {
                         returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, msgListOK);
                         return;
                     } else {
                         response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                     }
                 } catch (Throwable e) {
                     final String error = String.format("do Message Filter Exception, ConsumerGroup: %s Topic: %s ",
                     requestHeader.getConsumerGroup(), requestHeader.getTopic());
                            log.error(error, e);
                     response.setCode(ResponseCode.SYSTEM_ERROR);
                     response.setRemark(error + RemotingHelper.exceptionSimpleDesc(e));
                     returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
                     return;
                 }

                 break;
            case NO_MATCHED_MSG:
                 response.setCode(ResponseCode.PULL_RETRY_IMMEDIATELY);
                 break;
            case NO_NEW_MSG:
                 response.setCode(ResponseCode.PULL_NOT_FOUND);
                 break;
            case OFFSET_ILLEGAL:
                 response.setCode(ResponseCode.PULL_OFFSET_MOVED);
                 break;
            default:
                 break;
       }

       returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
  }

  @Override
  public void onException(Throwable e) {
      response.setCode(ResponseCode.SYSTEM_ERROR);
      response.setRemark("Pull Callback Exception, " + RemotingHelper.exceptionSimpleDesc(e));
      returnResponse(requestHeader.getConsumerGroup(), requestHeader.getTopic(), ctx, response, null);
      return;
   }
   };

   // 拉取消息
   pullConsumer.pullBlockIfNotFound(mq, null, offset, maxNums, pullCallback);
   return null;
}

在filtersrv中，也会有一个consumer，它的作用是从broker拉取消息，拉取到消息后，会先通过过滤类进行match，如果能够match成功，返回拉取到的消息。如果match失败，会通知consumer立即下一次拉取。

非常重要的一点：filtersrv拉取的消息也属于消费，过滤掉的消息不会返回到consumer，但其他的消费者也无法再次消费到。

4、consumer从filtersrv拉取消息，从broker对应的filtersrv列表中随机选取一个拉取消息，如果选取不到filtersrv，无法拉取到消息。所以，filtersrv一定需要高可用，其方式是一个broker可以对应有多个filtersrv，并且每个filtersrv会定期向broker发送心跳包，注册自己。