Apollo源码解析——配置发布

---

一、概述

配置发布流程

二、页面流程

这里我们提交新增的配置

会请求后台的release接口

2.代码流程

1. 看下portal模块下的ReleaseController#createRelease方法

这里首先做操作校验，然后调用releaseService.publish方法发布配置，最后广播出去事件

   @PreAuthorize(value = "@permissionValidator.hasReleaseNamespacePermission(#appId, #namespaceName, #env)")
  @PostMapping(value = "/apps/{appId}/envs/{env}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
  public ReleaseDTO createRelease(@PathVariable String appId,
                                  @PathVariable String env, @PathVariable String clusterName,
                                  @PathVariable String namespaceName, @RequestBody NamespaceReleaseModel model) {
    model.setAppId(appId);
    model.setEnv(env);
    model.setClusterName(clusterName);
    model.setNamespaceName(namespaceName);

    //若是紧急发布，但是当前环境未允许该操作，抛出异常
    if (model.isEmergencyPublish() && !portalConfig.isEmergencyPublishAllowed(Env.valueOf(env))) {
      throw new BadRequestException(String.format("Env: %s is not supported emergency publish now", env));
    }

    //发布配置
    ReleaseDTO createdRelease = releaseService.publish(model);

    //创建 ConfigPublishEvent 对象
    ConfigPublishEvent event = ConfigPublishEvent.instance();
    event.withAppId(appId)
        .withCluster(clusterName)
        .withNamespace(namespaceName)
        .withReleaseId(createdRelease.getId())
        .setNormalPublishEvent(true)
        .setEnv(Env.valueOf(env));

    //发布 ConfigPublishEvent 对象
    publisher.publishEvent(event);

    return createdRelease;
  }

2. ReleaseService#publish方法

这里发现直接通过http调用AdminService来发布配置项

  public ReleaseDTO publish(NamespaceReleaseModel model) {
    Env env = model.getEnv();
    boolean isEmergencyPublish = model.isEmergencyPublish();
    String appId = model.getAppId();
    String clusterName = model.getClusterName();
    String namespaceName = model.getNamespaceName();
    String releaseBy = StringUtils.isEmpty(model.getReleasedBy()) ?
                       userInfoHolder.getUser().getUserId() : model.getReleasedBy();

    // 调用 Admin Service API， 发布 Namespace 的配置
    ReleaseDTO releaseDTO = releaseAPI.createRelease(appId, env, clusterName, namespaceName,
                                                     model.getReleaseTitle(), model.getReleaseComment(),
                                                     releaseBy, isEmergencyPublish);

    Tracer.logEvent(TracerEventType.RELEASE_NAMESPACE,
                    String.format("%s+%s+%s+%s", appId, env, clusterName, namespaceName));

    return releaseDTO;
  }

3. adminService模块下的ReleaseController#publish方法

这里重点就是调用releaseService.publish发布配置，然后判断若有父 Namespace 对象，说明是子 Namespace ( 灰度发布 )，则使用父 Namespace 的Cluster名字，最后发送Release消息

  @Transactional
  @PostMapping("/apps/{appId}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
  public ReleaseDTO publish(@PathVariable("appId") String appId,
                            @PathVariable("clusterName") String clusterName,
                            @PathVariable("namespaceName") String namespaceName,
                            @RequestParam("name") String releaseName,
                            @RequestParam(name = "comment", required = false) String releaseComment,
                            @RequestParam("operator") String operator,
                            @RequestParam(name = "isEmergencyPublish", defaultValue = "false") boolean isEmergencyPublish) {
    // 检验对应的 Namespace 对象是否存在 若不存在抛出异常
    Namespace namespace = namespaceService.findOne(appId, clusterName, namespaceName);
    if (namespace == null) {
      throw new NotFoundException(String.format("Could not find namespace for %s %s %s", appId,
                                                clusterName, namespaceName));
    }
    // 发布 Namespace 的配置
    Release release = releaseService.publish(namespace, releaseName, releaseComment, operator, isEmergencyPublish);

    //获得 Cluster 名
    Namespace parentNamespace = namespaceService.findParentNamespace(namespace);
    String messageCluster;
    if (parentNamespace != null) {// 灰度发布
      messageCluster = parentNamespace.getClusterName();
    } else {
      messageCluster = clusterName; // 使用请求的 ClusterName
    }
    //发送 Release 消息
    messageSender.sendMessage(ReleaseMessageKeyGenerator.generate(appId, messageCluster, namespaceName),
            Topics.APOLLO_RELEASE_TOPIC);
    //将 Release 转换成 ReleaseDTO对象
    return BeanUtils.transform(ReleaseDTO.class, release);
  }

4. releaseService.publish方法

这里首先进行权限校验，这里会判断有没有父Namespace来判断是不是灰度发布，然后调用masterRelease主干发布配置，最后判断若有子namespace，调用mergeFromMasterAndPublishBranch合并主干并自己子namespace的发布。

  public Release publish(Namespace namespace, String releaseName, String releaseComment,
                         String operator, boolean isEmergencyPublish) {

    //发布配置时，会校验锁定人是否是当前的管理员
    checkLock(namespace, isEmergencyPublish, operator);

    //获得 Namespace 的普通配置 Map
    Map<String, String> operateNamespaceItems = getNamespaceItems(namespace);

    //获得 父 Namespace
    Namespace parentNamespace = namespaceService.findParentNamespace(namespace);

    //branch release
    //若有 父 Namespace，则是子 Namespace，进行灰度发布
    if (parentNamespace != null) {
      return publishBranchNamespace(parentNamespace, namespace, operateNamespaceItems,
                                    releaseName, releaseComment, operator, isEmergencyPublish);
    }

    //获得子 Namespace 对象
    Namespace childNamespace = namespaceService.findChildNamespace(namespace);

    //获得上一次，并且有效的 Release 对象
    Release previousRelease = null;
    if (childNamespace != null) {
      previousRelease = findLatestActiveRelease(namespace);
    }

    //master release
    //创建操作 Context
    Map<String, Object> operationContext = Maps.newLinkedHashMap();
    operationContext.put(ReleaseOperationContext.IS_EMERGENCY_PUBLISH, isEmergencyPublish);

    //主干发布
    Release release = masterRelease(namespace, releaseName, releaseComment, operateNamespaceItems,
                                    operator, ReleaseOperation.NORMAL_RELEASE, operationContext);

    //merge to branch and auto release
    //若有子 Namespace 时， 自动将主干合并到子 Namespace，并进行一次子 Namespace 的发布
    if (childNamespace != null) {
      mergeFromMasterAndPublishBranch(namespace, childNamespace, operateNamespaceItems,
                                      releaseName, releaseComment, operator, previousRelease,
                                      release, isEmergencyPublish);
    }

    return release;
  }

5. masterRelease方法

这里会新生成一个Release对象并保存，并且创建ReleaseHistory对象并保存，记录了当前的releaseId和它前一个版本的releaseId。

  private Release masterRelease(Namespace namespace, String releaseName, String releaseComment,
                                Map<String, String> configurations, String operator,
                                int releaseOperation, Map<String, Object> operationContext) {
    // 获得最后有效的 Release 对象
    Release lastActiveRelease = findLatestActiveRelease(namespace);
    long previousReleaseId = lastActiveRelease == null ? 0 : lastActiveRelease.getId();
    //创建 Release 对象，并保存
    Release release = createRelease(namespace, releaseName, releaseComment,
            configurations, operator);

    // 创建 ReleaseHistory 对象并保存
    releaseHistoryService.createReleaseHistory(namespace.getAppId(), namespace.getClusterName(),
            namespace.getNamespaceName(), namespace.getClusterName(),
            release.getId(), previousReleaseId, releaseOperation,
            operationContext, operator);

    return release;
  }

6. 这里已经发布完成后，会调用 messageSender.sendMessage发送出去一个Release消息

这里是把这条消息写到表里面了，并且把消息的Id添加到一个清理队列中，我们看什么地方会从队列中读取消息。

  public void sendMessage(String message, String channel) {
    logger.info("Sending message {} to channel {}", message, channel);
    if (!Objects.equals(channel, Topics.APOLLO_RELEASE_TOPIC)) {
      logger.warn("Channel {} not supported by DatabaseMessageSender!", channel);
      return;
    }

    Tracer.logEvent("Apollo.AdminService.ReleaseMessage", message);
    Transaction transaction = Tracer.newTransaction("Apollo.AdminService", "sendMessage");
    try {
      //保存 ReleaseMessage 对象
      ReleaseMessage newMessage = releaseMessageRepository.save(new ReleaseMessage(message));
      //添加到清理 Message 队列，若队列已满，添加失败，不阻塞等待
      toClean.offer(newMessage.getId());
      transaction.setStatus(Transaction.SUCCESS);
    } catch (Throwable ex) {
      logger.error("Sending message to database failed", ex);
      transaction.setStatus(ex);
      throw ex;
    } finally {
      transaction.complete();
    }
  }

7. DatabaseMessageSender#initialize方法

这里发现有一个线程池，并且提交了一个定时任务，会定时poll队列中的数据，然后调用cleanMessage方法

    cleanExecutorService.submit(() -> {
      // 若未停止，持续运行
      while (!cleanStopped.get() && !Thread.currentThread().isInterrupted()) {
        try {
          //拉取
          Long rm = toClean.poll(1, TimeUnit.SECONDS);
          //队列非空，处理拉取到的消息
          if (rm != null) {
            cleanMessage(rm);
          } else {
            //队列为空，sleep，避免空跑，占用CPU
            TimeUnit.SECONDS.sleep(5);
          }
        } catch (Throwable ex) {
          Tracer.logError(ex);
        }
      }
    });

8. DatabaseMessageSender#cleanMessage方法

这里发现其实这里并不是消息的消费者，这里的定时任务只是解决消息存在多个版本，只保留最新的版本。那我们去看一下，我们之前是把消息落到库里面了，看有没有线程定时去扫库。

   //double check in case the release message is rolled back
    // /查询对应的 ReleaseMessage 对象，避免已经删除，因为，DatabaseMessageSender 会在多进程中进行
    //会在多进程中执行。例如：1）Config Service + Admin Service ；2）N * Config Service ；3）N * Admin Service
    ReleaseMessage releaseMessage = releaseMessageRepository.findById(id).orElse(null);
    if (releaseMessage == null) {
      return;
    }
    boolean hasMore = true;
    //循环删除相同消息内容(`message`)的老消息
    while (hasMore && !Thread.currentThread().isInterrupted()) {
      //拉取相同消息内容的100条老消息
      //老消息的定义：比当前消息编号小，即先发送的
      //按照 id 升序
      List<ReleaseMessage> messages = releaseMessageRepository.findFirst100ByMessageAndIdLessThanOrderByIdAsc(
              releaseMessage.getMessage(), releaseMessage.getId());

      //删除老消息
      releaseMessageRepository.deleteAll(messages);
      // 若拉取不足 100 条，说明无老消息
      hasMore = messages.size() == 100;

      messages.forEach(toRemove -> Tracer.logEvent(
              String.format("ReleaseMessage.Clean.%s", toRemove.getMessage()), String.valueOf(toRemove.getId())));
    }

9. ReleaseMessageScanner#afterPropertiesSet方法

根据我们上面的判断，发现在ReleaseMessageScanner中的初始化方法中会启动一个定时任务定期去扫库

  public void afterPropertiesSet() throws Exception {
    //从 ServerConfig 中获得频率
    databaseScanInterval = bizConfig.releaseMessageScanIntervalInMilli();
    //获得 最大的 ReleaseMessage 的编号
    maxIdScanned = loadLargestMessageId();
    // 创建从 DB 中扫描 ReleaseMessage 表的定时任务
    executorService.scheduleWithFixedDelay(() -> {
      Transaction transaction = Tracer.newTransaction("Apollo.ReleaseMessageScanner", "scanMessage");
      try {
        //从 DB 中，扫描 ReleaseMessage 们
        scanMissingMessages();
        scanMessages();
        transaction.setStatus(Transaction.SUCCESS);
      } catch (Throwable ex) {
        transaction.setStatus(ex);
        logger.error("Scan and send message failed", ex);
      } finally {
        transaction.complete();
      }
    }, databaseScanInterval, databaseScanInterval, TimeUnit.MILLISECONDS);

  }

10. scanMessages方法

继续看scanAndSendMessages方法

  private void scanMessages() {
    boolean hasMoreMessages = true;
    while (hasMoreMessages && !Thread.currentThread().isInterrupted()) {
      hasMoreMessages = scanAndSendMessages();
    }
  }

11. scanAndSendMessages方法

这里的代码就是拉取消息，然后触发监听这个消息的监听器，把消息传递出去。

  private boolean scanAndSendMessages() {
    //current batch is 500
    //获得大于 maxIdScanned 的 500 条 ReleaseMessage 记录，按照 id 升序
    List<ReleaseMessage> releaseMessages =
        releaseMessageRepository.findFirst500ByIdGreaterThanOrderByIdAsc(maxIdScanned);
    if (CollectionUtils.isEmpty(releaseMessages)) {
      return false;
    }
    //触发监听器
    fireMessageScanned(releaseMessages);
    //获得新的maxIdScanned，取最后一条记录
    int messageScanned = releaseMessages.size();
    long newMaxIdScanned = releaseMessages.get(messageScanned - 1).getId();
    // check id gaps, possible reasons are release message not committed yet or already rolled back
    if (newMaxIdScanned - maxIdScanned > messageScanned) {
      recordMissingReleaseMessageIds(releaseMessages, maxIdScanned);
    }
    maxIdScanned = newMaxIdScanned;
    //若拉取不足 500  条，说明无新消息了
    return messageScanned == 500;
  }

12. fireMessageScanned 方法

这里就是通知所有的监听器，我们看下这些监听器是什么时候注册的。

 private void fireMessageScanned(Iterable<ReleaseMessage> messages) {
    for (ReleaseMessage message : messages) { //循环 ReleaseMessage
      for (ReleaseMessageListener listener : listeners) {
        try {
          //触发监听器
          listener.handleMessage(message, Topics.APOLLO_RELEASE_TOPIC);
        } catch (Throwable ex) {
          Tracer.logError(ex);
          logger.error("Failed to invoke message listener {}", listener.getClass(), ex);
        }
      }
    }
  }

13. ReleaseMessageScanner#addMessageListener方法

我们看到是通过这个方法添加监听器的，我们继续追溯它的调度方

  public void addMessageListener(ReleaseMessageListener listener) {
    if (!listeners.contains(listener)) {
      listeners.add(listener);
    }
  }

14. ConfigServiceAutoConfiguration#releaseMessageScanner方法

我们看到是在这个地方注册对应的监听器，这里我们主要看客户端的通知，主要看notificationControllerV2

    @Bean
    public ReleaseMessageScanner releaseMessageScanner() {
      ReleaseMessageScanner releaseMessageScanner = new ReleaseMessageScanner();
      //0. handle release message cache
      releaseMessageScanner.addMessageListener(releaseMessageServiceWithCache);
      //1. handle gray release rule
      releaseMessageScanner.addMessageListener(grayReleaseRulesHolder);
      //2. handle server cache
      releaseMessageScanner.addMessageListener(configService);
      releaseMessageScanner.addMessageListener(configFileController);
      //3. notify clients
      releaseMessageScanner.addMessageListener(notificationControllerV2);
      releaseMessageScanner.addMessageListener(notificationController);
      return releaseMessageScanner;
    }
  }

15. notificationControllerV2#handleMessage方法

这里接到消息过后会通知对应相关的客户端，这块代码之后再详细解析

    public void handleMessage(ReleaseMessage message, String channel) {
        logger.info("message received - channel: {}, message: {}", channel, message);

        String content = message.getMessage();
        Tracer.logEvent("Apollo.LongPoll.Messages", content);
        // 仅处理 APOLLO_RELEASE_TOPIC
        if (!Topics.APOLLO_RELEASE_TOPIC.equals(channel) || Strings.isNullOrEmpty(content)) {
            return;
        }

        // 获得对应的 Namespace 名字
        String changedNamespace = retrieveNamespaceFromReleaseMessage.apply(content);

        if (Strings.isNullOrEmpty(changedNamespace)) {
            logger.error("message format invalid - {}", content);
            return;
        }

        if (!deferredResults.containsKey(content)) {
            return;
        }

        //create a new list to avoid ConcurrentModificationException
        // 创建 DeferredResultWrapper 数组，避免并发问题
        List<DeferredResultWrapper> results = Lists.newArrayList(deferredResults.get(content));

        // 创建 ApolloConfigNotification 对象
        ApolloConfigNotification configNotification = new ApolloConfigNotification(changedNamespace, message.getId());
        configNotification.addMessage(content, message.getId());

        //do async notification if too many clients
        // 若需要通知的客户端过多，使用 ExecutorService 异步通知，避免`惊群效应`
        if (results.size() > bizConfig.releaseMessageNotificationBatch()) {
            largeNotificationBatchExecutorService.submit(() -> {
                logger.debug("Async notify {} clients for key {} with batch {}", results.size(), content,
                        bizConfig.releaseMessageNotificationBatch());
                for (int i = 0; i < results.size(); i++) {
                    // 每 N 个K客户端，Sleep 一段事件
                    if (i > 0 && i % bizConfig.releaseMessageNotificationBatch() == 0) {
                        try {
                            TimeUnit.MILLISECONDS.sleep(bizConfig.releaseMessageNotificationBatchIntervalInMilli());
                        } catch (InterruptedException e) {
                            //ignore
                        }
                    }
                    logger.debug("Async notify {}", results.get(i));
                    // 设置结果
                    results.get(i).setResult(configNotification);
                }
            });
            return;
        }

        logger.debug("Notify {} clients for key {}", results.size(), content);

        for (DeferredResultWrapper result : results) {
            result.setResult(configNotification);
        }
        logger.debug("Notification completed");
    }

总结

这里配置的更新即通知相应关心的客户端的实现方式如下

1. Admin Service 在配置发布后会往 ReleaseMessage 表插入一条消息记录，消息内容就是配置发布的 AppId+Cluster+Namespace ，参见 DatabaseMessageSender 。
2. Config Service 有一个线程会每秒扫描一次 ReleaseMessage 表，看看是否有新的消息记录，参见ReleaseMessageScanner。
3. Config Service 如果发现有新的消息记录，那么就会通知到所有的消息监听器（ReleaseMessageListener），如 NotificationControllerV2 ，消息监听器的注册过程参见 ConfigServiceAutoConfiguration 。
4. NotificationControllerV2 得到配置发布的 AppId+Cluster+Namespace 后，会通知对应的客户端。