soul-admin和soul-bootstrap使用http长轮询同步数据源码分析
soul-admin 端
- 开启配置
- 加载配置
启动时会注册一个HttpLongPollingDataChangedListener
用来接收soul-bootstrap发来的获取最新配置还有定时更新本地缓存的数据
查看HttpLongPollingDataChangedListener
的构造方法
public HttpLongPollingDataChangedListener(final HttpSyncProperties httpSyncProperties) {
// 创建一个阻塞队列
this.clients = new ArrayBlockingQueue<>(1024);
// 创建一个定时线程池
this.scheduler = new ScheduledThreadPoolExecutor(1,
SoulThreadFactory.create("long-polling", true));
// 长轮询的配置信息, 主要是轮询的时间间隔,还有轮询功能是否开启
this.httpSyncProperties = httpSyncProperties;
}
该类实现了InitializingBean.afterInitialize
方法,方法内启动了一个定时线程池,定时在从数据库内获取全量的配置数据,与当前内存中的数据比对,如果不同则更新内存中的数据
protected void afterInitialize() {
long syncInterval = httpSyncProperties.getRefreshInterval().toMillis();
// Periodically check the data for changes and update the cache 创建定时线程池,每5分钟从数据库内获取全量数据
scheduler.scheduleWithFixedDelay(() -> {
log.info("http sync strategy refresh config start.");
try {
// 从数据库内获取全量数据
this.refreshLocalCache();
log.info("http sync strategy refresh config success.");
} catch (Exception e) {
log.error("http sync strategy refresh config error!", e);
}
}, syncInterval, syncInterval, TimeUnit.MILLISECONDS);
log.info("http sync strategy refresh interval: {}ms", syncInterval);
}
// 更新数据
private void refreshLocalCache() {
this.updateAppAuthCache();
this.updatePluginCache();
this.updateRuleCache();
this.updateSelectorCache();
this.updateMetaDataCache();
}
soul-admin端提供了两个接口
/configs/fetch
根据group获取本地的缓存配置数据
/configs/listener
获取数据接口,根据调用方传递过来的group数据的md5和lastModifyTime与本地缓存的数据进行对比,获取发生改变了的group,如果没有改变则将请求放入队列中,等待group改变时返回修改的数据。
/configs/listener 接口核心代码
public void doLongPolling(final HttpServletRequest request, final HttpServletResponse response) {
// compare group md5 比较数据的md5值 和 lastModifyTime 获取改变了的 group
List<ConfigGroupEnum> changedGroup = compareChangedGroup(request);
String clientIp = getRemoteIp(request);
// response immediately. 如果有改变group的数据立即返回
if (CollectionUtils.isNotEmpty(changedGroup)) {
this.generateResponse(response, changedGroup);
log.info("send response with the changed group, ip={}, group={}", clientIp, changedGroup);
return;
}
// listen for configuration changed.
final AsyncContext asyncContext = request.startAsync();
// AsyncContext.settimeout() does not timeout properly, so you have to control it yourself
asyncContext.setTimeout(0L);
// block client's thread. 将当前请求放入到定时线程池内
scheduler.execute(new LongPollingClient(asyncContext, clientIp, HttpConstants.SERVER_MAX_HOLD_TIMEOUT));
}
看下LongPollingClient
内的方法
public void run() {
// 任务放入到 线程池中,并且设置延迟执行
this.asyncTimeoutFuture = scheduler.schedule(() -> {
// 队列中的相同类型的任务(我的理解是因为新请求之前的请求也没有用了,使用最新的即可)
clients.remove(LongPollingClient.this);
// 比较数据的md5值 和 lastModifyTime 查看数据是否改变
List<ConfigGroupEnum> changedGroups = compareChangedGroup((HttpServletRequest) asyncContext.getRequest());
// 返回响应
sendResponse(changedGroups);
}, timeoutTime, TimeUnit.MILLISECONDS);
// 将当前任务放入到阻塞队列中,当admin端改变数据时会调用立即从当前队列中拿出该任务返回修改的group,
// 并且取消当前任务的定时调度,即DataChangeTask任务
clients.add(this);
}
每次admin修改数据时都会通过事件机制(与其他数据同步一样)调用线程池调用DataChangeTask
任务
protected void afterSelectorChanged(final List<SelectorData> changed, final DataEventTypeEnum eventType) {
scheduler.execute(new DataChangeTask(ConfigGroupEnum.SELECTOR));
}
public void run() {
// 循环 调用队列中的任务
for (Iterator<LongPollingClient> iter = clients.iterator(); iter.hasNext();) {
LongPollingClient client = iter.next();
iter.remove();
client.sendResponse(Collections.singletonList(groupKey));
log.info("send response with the changed group,ip={}, group={}, changeTime={}", client.ip, groupKey, changeTime);
}
}
void sendResponse(final List<ConfigGroupEnum> changedGroups) {
// cancel scheduler 取消之前设置的延迟任务
if (null != asyncTimeoutFuture) {
asyncTimeoutFuture.cancel(false);
}
generateResponse((HttpServletResponse) asyncContext.getResponse(), changedGroups);
asyncContext.complete();
}
soul-bootstrap端
- 开启配置
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-fk40vO2w-1611490071459)(https://uploader.shimo.im/f/pZBD4KcV5iuVi0Ic.png!thumbnail?fileGuid=9k3gwDh8Q6Gpc9ky)]
- 加载配置
启动时会注册一个HttpSyncDataService
用来拉取soul-admin端的数据,即定时调用 admin端的/configs/listener
接口获取数据
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-SFK5Gsq3-1611490071461)(https://uploader.shimo.im/f/0jvLz8CTKAtK4PmO.png!thumbnail?fileGuid=9k3gwDh8Q6Gpc9ky)]
HttpSyncDataService
源码
public HttpSyncDataService(final HttpConfig httpConfig, final PluginDataSubscriber pluginDataSubscriber,
final List<MetaDataSubscriber> metaDataSubscribers, final List<AuthDataSubscriber> authDataSubscribers) {
// 设置更新不同类型数据的 Subscriber
this.factory = new DataRefreshFactory(pluginDataSubscriber, metaDataSubscribers, authDataSubscribers);
// soul-admin 的配置
this.httpConfig = httpConfig;
// soul-admin 地址
this.serverList = Lists.newArrayList(Splitter.on(",").split(httpConfig.getUrl()));
// 发送http请求用
this.httpClient = createRestTemplate();
// 调用获取要同步的配置数据
this.start();
}
private void start() {
// It could be initialized multiple times, so you need to control that.
if (RUNNING.compareAndSet(false, true)) {
// fetch all group configs. 抓取全部配置
this.fetchGroupConfig(ConfigGroupEnum.values());
int threadSize = serverList.size();
// 创建线程池
this.executor = new ThreadPoolExecutor(threadSize, threadSize, 60L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(),
SoulThreadFactory.create("http-long-polling", true));
// start long polling, each server creates a thread to listen for changes.
// 为每一个配置的 soul-admin执行一个任务
this.serverList.forEach(server -> this.executor.execute(new HttpLongPollingTask(server)));
} else {
log.info("soul http long polling was started, executor=[{}]", executor);
}
}
HttpLongPollingTask
public void run() {
// 循环 一直轮询
while (RUNNING.get()) {
for (int time = 1; time <= retryTimes; time++) {
try {
// 发送请求 更新数据
doLongPolling(server);
} catch (Exception e) {
// print warnning log.
if (time < retryTimes) {
log.warn("Long polling failed, tried {} times, {} times left, will be suspended for a while! {}",
time, retryTimes - time, e.getMessage());
ThreadUtils.sleep(TimeUnit.SECONDS, 5);
continue;
}
// print error, then suspended for a while.
log.error("Long polling failed, try again after 5 minutes!", e);
ThreadUtils.sleep(TimeUnit.MINUTES, 5);
}
}
}
log.warn("Stop http long polling.");
}
private void doLongPolling(final String server) {
MultiValueMap<String, String> params = new LinkedMultiValueMap<>(8);
// 所有 group
for (ConfigGroupEnum group : ConfigGroupEnum.values()) {
ConfigData<?> cacheConfig = factory.cacheConfigData(group);
String value = String.join(",", cacheConfig.getMd5(), String.valueOf(cacheConfig.getLastModifyTime()));
params.put(group.name(), Lists.newArrayList(value));
}
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.APPLICATION_FORM_URLENCODED);
HttpEntity httpEntity = new HttpEntity(params, headers);
String listenerUrl = server + "/configs/listener";
log.debug("request listener configs: [{}]", listenerUrl);
JsonArray groupJson = null;
try {
// 调用soul-admin configs/listener 接口获取所有修改过的 group
String json = this.httpClient.postForEntity(listenerUrl, httpEntity, String.class).getBody();
log.debug("listener result: [{}]", json);
groupJson = GSON.fromJson(json, JsonObject.class).getAsJsonArray("data");
} catch (RestClientException e) {
String message = String.format("listener configs fail, server:[%s], %s", server, e.getMessage());
throw new SoulException(message, e);
}
if (groupJson != null) {
// fetch group configuration async. 如果有修改过的 group 则调用 /configs/fetch 接口获取具体修改的数据
ConfigGroupEnum[] changedGroups = GSON.fromJson(groupJson, ConfigGroupEnum[].class);
if (ArrayUtils.isNotEmpty(changedGroups)) {
log.info("Group config changed: {}", Arrays.toString(changedGroups));
// 调用 /configs/fetch 接口获取具体修改的数据
this.doFetchGroupConfig(server, changedGroups);
}
}
}
以上就是 使用http长轮询同步配置的源码解读