Soul 网关源码阅读之 http 长轮询同步方式

除去引入中间件的方式，Soul 网关还提供了 http 长轮询的方式进行数据的同步。
和其他方式一样，需要在 bootstrap 的 pom.xml 文件中引入如下依赖：

<!--soul data sync start use http-->
     <dependency>
           <groupId>org.dromara</groupId>
           <artifactId>soul-spring-boot-starter-sync-data-http</artifactId>
           <version>${last.version}</version>
     </dependency>

在 yml 文件中进行如下配置：

soul :
   sync:
       http:
            url: http://localhost:9095
#url: 配置成你的 soul-admin的 ip与端口地址，多个admin集群环境请使用（,）分隔。

在 admin 模块的 yml 文件中进行下述配置：

soul:
  sync:
     http:
     	enabled: true

HttpSyncDataService

由于长轮询的请求发起方是 bootstrap 模块，所以本篇文章从该模块开始进行 http 长轮询的源码阅读。
首先是 org.dromara.soul.sync.data.http.HttpSyncDataService 的构造方法：

	public HttpSyncDataService(final HttpConfig httpConfig, final PluginDataSubscriber pluginDataSubscriber,
                               final List<MetaDataSubscriber> metaDataSubscribers, final List<AuthDataSubscriber> authDataSubscribers) {
        this.factory = new DataRefreshFactory(pluginDataSubscriber, metaDataSubscribers, authDataSubscribers);
        this.httpConfig = httpConfig;
        // 初始化服务端路由列表，以此支持多实例 admin
        this.serverList = Lists.newArrayList(Splitter.on(",").split(httpConfig.getUrl()));
        // http 客户端初始化
        this.httpClient = createRestTemplate();
        this.start();
    }

Soul 使用了 OkHttp 进行 http 请求：

	private RestTemplate createRestTemplate() {
        OkHttp3ClientHttpRequestFactory factory = new OkHttp3ClientHttpRequestFactory();
        factory.setConnectTimeout((int) this.connectionTimeout.toMillis());
        factory.setReadTimeout((int) HttpConstants.CLIENT_POLLING_READ_TIMEOUT);
        return new RestTemplate(factory);
    }

构造方法最终执行了 start 方法，内容如下：

private void start() {
        // It could be initialized multiple times, so you need to control that.
        // 防止多线程场景下多次调用的一致性问题
        if (RUNNING.compareAndSet(false, true)) {
            // fetch all group configs.
            // 全量的拉取所有配置
            this.fetchGroupConfig(ConfigGroupEnum.values());
            int threadSize = serverList.size();
            // 初始化一个线程池，用于轮询线程的管理
            this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
                    new LinkedBlockingQueue<>(),
                    SoulThreadFactory.create("http-long-polling", true));
            // start long polling, each server creates a thread to listen for changes.
            // 给每一个服务地址开启一个任务
            this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
        } else {
            log.info("soul http long polling was started, executor=[{}]", executor);
        }
    }

进一步，我们查看拉取配置的方法

    private void fetchGroupConfig(final ConfigGroupEnum... groups) throws SoulException {
        for (int index = 0; index < this.serverList.size(); index++) {
            String server = serverList.get(index);
            try {
                this.doFetchGroupConfig(server, groups);
                break;
            } catch (SoulException e) {
                // no available server, throw exception.
                if (index >= serverList.size() - 1) {
                    throw e;
                }
                log.warn("fetch config fail, try another one: {}", serverList.get(index + 1));
            }
        }
    }
    
    private void doFetchGroupConfig(final String server, final ConfigGroupEnum... groups) {
        StringBuilder params = new StringBuilder();
        // 组装 URL
        for (ConfigGroupEnum groupKey : groups) {
            params.append("groupKeys").append("=").append(groupKey.name()).append("&");
        }
        String url = server + "/configs/fetch?" + StringUtils.removeEnd(params.toString(), "&");
        log.info("request configs: [{}]", url);
        String json = null;
        try {
            // 使用上文中初始化的 HTTP 客户端进行请求
            json = this.httpClient.getForObject(url, String.class);
        } catch (RestClientException e) {
            String message = String.format("fetch config fail from server[%s], %s", url, e.getMessage());
            log.warn(message);
            throw new SoulException(message, e);
        }
        // update local cache
        // 拿到结果后，更新本地缓存
        boolean updated = this.updateCacheWithJson(json);
        if (updated) {
            log.info("get latest configs: [{}]", json);
            return;
        }
        // not updated. it is likely that the current config server has not been updated yet. wait a moment.
        log.info("The config of the server[{}] has not been updated or is out of date. Wait for 30s to listen for changes again.", server);
        ThreadUtils.sleep(TimeUnit.SECONDS, 30);
    }

接下来，我们回过头再阅读线程池中的长轮询任务：

        public void run() {
            while (RUNNING.get()) {
                // 按照配置，进行重试
                for (int time = 1; time <= retryTimes; time++) {
                    try {
                        doLongPolling(server);
                    } catch (Exception e) {
                        // print warnning log.
                        if (time < retryTimes) {
                            log.warn("Long polling failed, tried {} times, {} times left, will be suspended for a while! {}",
                                    time, retryTimes - time, e.getMessage());
                            ThreadUtils.sleep(TimeUnit.SECONDS, 5);
                            continue;
                        }
                        // print error, then suspended for a while.
                        log.error("Long polling failed, try again after 5 minutes!", e);
                        ThreadUtils.sleep(TimeUnit.MINUTES, 5);
                    }
                }
            }
            log.warn("Stop http long polling.");
        }
        
        private void doLongPolling(final String server) {
            MultiValueMap<String, String> params = new LinkedMultiValueMap<>(8);
            // 组装 URL
            for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
                ConfigData<?> cacheConfig = factory.cacheConfigData(group);
                String value = String.join(",", cacheConfig.getMd5(), String.valueOf(cacheConfig.getLastModifyTime()));
                params.put(group.name(), Lists.newArrayList(value));
            }
            HttpHeaders headers = new HttpHeaders();
            headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
            HttpEntity httpEntity = new HttpEntity(params, headers);
            // 请求的路径为 /configs/listener
            String listenerUrl = server + "/configs/listener";
            log.debug("request listener configs: [{}]", listenerUrl);
            JsonArray groupJson = null;
            try {
                String json = this.httpClient.postForEntity(listenerUrl, httpEntity, String.class).getBody();
                log.debug("listener result: [{}]", json);
                groupJson = GSON.fromJson(json, JsonObject.class).getAsJsonArray("data");
            } catch (RestClientException e) {
                String message = String.format("listener configs fail, server:[%s], %s", server, e.getMessage());
                throw new SoulException(message, e);
            }
            // 如果 listener 接口返回了数据，则重新获取相应组别的配置
            if (groupJson != null) {
                // fetch group configuration async.
                ConfigGroupEnum[] changedGroups = GSON.fromJson(groupJson, ConfigGroupEnum[].class);
                if (ArrayUtils.isNotEmpty(changedGroups)) {
                    log.info("Group config changed: {}", Arrays.toString(changedGroups));
                    this.doFetchGroupConfig(server, changedGroups);
                }
            }
        }

可以看到，轮询的任务会先调用 /configs/listener 接口获取哪些组别（比如 RULE）的配置发生了变化，当获取到变化时，再调用 /configs/fetch 拉取相应组别的配置，注意，此时是全量拉取。

ConfigController

在前文的探索中，可以发现 http 长轮询只涉及到两个接口，它们都在 org.dromara.soul.admin.controller.ConfigController 中被定义。而每个接口都依赖 org.dromara.soul.admin.listener.http.HttpLongPollingDataChangedListener

先从 /listener 接口开始看：

    @PostMapping(value = "/listener")
    public void listener(final HttpServletRequest request, final HttpServletResponse response) {
        longPollingListener.doLongPolling(request, response);
    }
    
    public void doLongPolling(final HttpServletRequest request, final HttpServletResponse response) {

        // compare group md5
        // 获取有变化的组
        List<ConfigGroupEnum> changedGroup = compareChangedGroup(request);
        String clientIp = getRemoteIp(request);

        // response immediately.
        if (CollectionUtils.isNotEmpty(changedGroup)) {
            this.generateResponse(response, changedGroup);
            log.info("send response with the changed group, ip={}, group={}", clientIp, changedGroup);
            return;
        }

        // listen for configuration changed.
        // 使用异步的方式进行请求处理
        final AsyncContext asyncContext = request.startAsync();

        // AsyncContext.settimeout() does not timeout properly, so you have to control it yourself
        asyncContext.setTimeout(0L);

        // block client's thread.
        // 将异步的上下文放入线程池，将在最多 60 秒的等待被执行
        scheduler.execute(new LongPollingClient(asyncContext, clientIp, HttpConstants.SERVER_MAX_HOLD_TIMEOUT));
    }

接下来需要注意的是，上述代码块中最后放入线程池的任务，会最终进入 HttpLongPollingDataChangedListener 维护的一个 BlockingQueue 中，以下是 LongPollingClient 的 run() 方法：

        @Override
        public void run() {
            // 生成一个定时执行的任务
            this.asyncTimeoutFuture = scheduler.schedule(() -> {
                clients.remove(LongPollingClient.this);
                List<ConfigGroupEnum> changedGroups = compareChangedGroup((HttpServletRequest) asyncContext.getRequest());
                sendResponse(changedGroups);
            }, timeoutTime, TimeUnit.MILLISECONDS);
            // 最终放入任务队列中
            clients.add(this);
        }

除去上述的执行方式，也有在配置修改时返回 response 的逻辑：
HttpLongPollingDataChangedListener 中也实现了 afterRuleChanged 方法，这个方法向线程池中放入了 DataChangeTask 任务：

        @Override
        public void run() {
            for (Iterator<LongPollingClient> iter = clients.iterator(); iter.hasNext();) {
                // 获取到一个正在等待响应的客户端线程
                LongPollingClient client = iter.next();
                iter.remove();
                // 发出响应
                client.sendResponse(Collections.singletonList(groupKey));
                log.info("send response with the changed group,ip={}, group={}, changeTime={}", client.ip, groupKey, changeTime);
            }
        }