1、Apollo架构设计
- Config Service提供配置的读取、推送等功能,服务对象是Apollo客户端
- Admin Service提供配置的修改、发布等功能,服务对象是Apollo Portal(管理界面)
- Config Service和Admin Service都是多实例、无状态部署,所以需要将自己注册到Eureka中并保持心跳
- 在Eureka之上我们架了一层Meta Server用于封装Eureka的服务发现接口
- Client通过域名访问Meta Server获取Config Service服务列表(IP+Port),而后直接通过IP+Port访问服务,同时在Client侧会做load balance、错误重试
- Portal通过域名访问Meta Server获取Admin Service服务列表(IP+Port),而后直接通过IP+Port访问服务,同时在Portal侧会做load balance、错误重试
- 为了简化部署,实际上Config Service、Eureka和Meta Server三个逻辑角色部署在同一个JVM进程中
2、配置发布后的实时推送
上图简要描述了配置发布的大致过程:
- 用户在Portal操作配置发布
- Portal调用Admin Service的接口操作发布
- Admin Service发布配置后,发送ReleaseMessage给各个Config Service
- Config Service收到ReleaseMessage后,通知对应的客户端
下面我们就结合上述流程来解析对应源码:
1)、Portal发布配置
Apollo Portal模块:
在Apollo点击发布按钮,调用com.ctrip.framework.apollo.portal.controller.ReleaseController
提供的API
@Validated
@RestController
public class ReleaseController {
@PreAuthorize(value = "@permissionValidator.hasReleaseNamespacePermission(#appId, #namespaceName, #env)")
@PostMapping(value = "/apps/{appId}/envs/{env}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
public ReleaseDTO createRelease(@PathVariable String appId,
@PathVariable String env, @PathVariable String clusterName,
@PathVariable String namespaceName, @RequestBody NamespaceReleaseModel model) {
model.setAppId(appId);
model.setEnv(env);
model.setClusterName(clusterName);
model.setNamespaceName(namespaceName);
if (model.isEmergencyPublish() && !portalConfig.isEmergencyPublishAllowed(Env.valueOf(env))) {
throw new BadRequestException(String.format("Env: %s is not supported emergency publish now", env));
}
//发布配置
ReleaseDTO createdRelease = releaseService.publish(model);
ConfigPublishEvent event = ConfigPublishEvent.instance();
event.withAppId(appId)
.withCluster(clusterName)
.withNamespace(namespaceName)
.withReleaseId(createdRelease.getId())
.setNormalPublishEvent(true)
.setEnv(Env.valueOf(env));
publisher.publishEvent(event);
return createdRelease;
}
这里核心代码是调用ReleaseService#publish(NamespaceReleaseModel)
方法,调用Admin Service API,发布配置,NamespaceReleaseModel
属性如下:
public class NamespaceReleaseModel implements Verifiable {
/**
* app编号
*/
private String appId;
/**
* env名称
*/
private String env;
/**
* cluster名称
*/
private String clusterName;
/**
* namespace名称
*/
private String namespaceName;
/**
* 发布标题
*/
private String releaseTitle;
/**
* 发布描述
*/
private String releaseComment;
/**
* 发布人
*/
private String releasedBy;
/**
* 是否紧急发布
*/
private boolean isEmergencyPublish;
接着来看ReleaseService#publish(NamespaceReleaseModel)
方法:
@Service
public class ReleaseService {
public ReleaseDTO publish(NamespaceReleaseModel model) {
Env env = model.getEnv();
boolean isEmergencyPublish = model.isEmergencyPublish();
String appId = model.getAppId();
String clusterName = model.getClusterName();
String namespaceName = model.getNamespaceName();
String releaseBy = StringUtils.isEmpty(model.getReleasedBy()) ?
userInfoHolder.getUser().getUserId() : model.getReleasedBy();
//调用Admin Service API,发布Namespace的配置
ReleaseDTO releaseDTO = releaseAPI.createRelease(appId, env, clusterName, namespaceName,
model.getReleaseTitle(), model.getReleaseComment(),
releaseBy, isEmergencyPublish);
Tracer.logEvent(TracerEventType.RELEASE_NAMESPACE,
String.format("%s+%s+%s+%s", appId, env, clusterName, namespaceName));
return releaseDTO;
}
ReleaseService#publish(NamespaceReleaseModel)
中调用ReleaseAPI#createRelease(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)
方法,调用Admin Service API,发布Namespace的配置
com.ctrip.framework.apollo.portal.api.ReleaseAPI
实现API抽象类,封装对Admin Service Release模块的API调用:
@Service
public class AdminServiceAPI {
@Service
public static class ReleaseAPI extends API {
public ReleaseDTO createRelease(String appId, Env env, String clusterName, String namespace,
String releaseName, String releaseComment, String operator,
boolean isEmergencyPublish) {
HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.parseMediaType(MediaType.APPLICATION_FORM_URLENCODED_VALUE + ";charset=UTF-8"));
MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
parameters.add("name", releaseName);
parameters.add("comment", releaseComment);
parameters.add("operator", operator);
parameters.add("isEmergencyPublish", String.valueOf(isEmergencyPublish));
HttpEntity<MultiValueMap<String, String>> entity =
new HttpEntity<>(parameters, headers);
ReleaseDTO response = restTemplate.post(
env, "apps/{appId}/clusters/{clusterName}/namespaces/{namespaceName}/releases", entity,
ReleaseDTO.class, appId, clusterName, namespace);
return response;
}
2)、Admin Service发送ReleaseMessage(异步)
Admin Service模块:
createRelease()
中使用RestTemplate调用ReleaseController#publish(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)
方法提供的API,发布Namespace的配置
@RestController
public class ReleaseController {
@Transactional
@PostMapping("/apps/{appId}/clusters/{clusterName}/namespaces/{namespaceName}/releases")
public ReleaseDTO publish(@PathVariable("appId") String appId,
@PathVariable("clusterName") String clusterName,
@PathVariable("namespaceName") String namespaceName,
@RequestParam("name") String releaseName,
@RequestParam(name = "comment", required = false) String releaseComment,
@RequestParam("operator") String operator,
@RequestParam(name = "isEmergencyPublish", defaultValue = "false") boolean isEmergencyPublish) {
//校验对应的Namespace是否存在
Namespace namespace = namespaceService.findOne(appId, clusterName, namespaceName);
if (namespace == null) {
throw new NotFoundException(String.format("Could not find namespace for %s %s %s", appId,
clusterName, namespaceName));
}
Release release = releaseService.publish(namespace, releaseName, releaseComment, operator, isEmergencyPublish);
Namespace parentNamespace = namespaceService.findParentNamespace(namespace);
String messageCluster;
if (parentNamespace != null) {
messageCluster = parentNamespace.getClusterName();
} else {
messageCluster = clusterName;
}
//发送Release消息
messageSender.sendMessage(ReleaseMessageKeyGenerator.generate(appId, messageCluster, namespaceName),
Topics.APOLLO_RELEASE_TOPIC);
return BeanUtils.transform(ReleaseDTO.class, release);
}
发送ReleaseMessage的实现方式:
Admin Service在配置发布后,需要通知所有的Config Service有配置发布,从而Config Service可以通知对应的客户端来拉取最新的配置
这是一个典型的消息使用场景,Admin Service作为producer发出消息,各个Config Service作为consumer消费消息。通过一个消息组件(Message Queue)就能很好的实现Admin Service和Config Service的解耦
Apollo没有采用外部的消息中间件,而是通过数据库实现了一个简单的消息队列,实现方式如下:
- Admin Service在配置发布后会往ReleaseMessage表插入一条消息记录,消息内容就是配置发布的AppId+Cluster+Namespace,参见DatabaseMessageSender
- Config Service有一个线程会每秒扫描一次ReleaseMessage表,看看是否有新的消息记录,参见ReleaseMessageScanner
- Config Service如果发现有新的消息记录,那么就会通知到所有的消息监听器(ReleaseMessageListener),如NotificationControllerV2,消息监听器的注册过程参见ConfigServiceAutoConfiguration
- NotificationControllerV2得到配置发布的AppId+Cluster+Namespace后,会通知对应的客户端
ReleaseMessage:
@Entity
@Table(name = "ReleaseMessage")
public class ReleaseMessage {
/**
* id自增主键
*/
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "Id")
private long id;
/**
* 消息内容,通过ReleaseMessageKeyGenerator.generate()生成,appId+cluster+namespace
*/
@Column(name = "Message", nullable = false)
private String message;
/**
* 最后更新时间
*/
@Column(name = "DataChange_LastTime")
private Date dataChangeLastModifiedTime;
ReleaseMessageKeyGenerator:
public class ReleaseMessageKeyGenerator {
private static final Joiner STRING_JOINER = Joiner.on(ConfigConsts.CLUSTER_NAMESPACE_SEPARATOR);
public static String generate(String appId, String cluster, String namespace) {
//将appId+cluster+namespace拼接,使用+作为间隔
return STRING_JOINER.join(appId, cluster, namespace);
}
}
将appId+cluster+namespace拼接,使用+作为间隔,因此对于同一个Namespace,生成的消息内容是相同的。通过这样的方式,可以使用最新的ReleaseMessage的id属性,作为Namespace是否发生变更的标识。而Apollo确实是通过这样的方式实现,Client通过不断使用获得到ReleaseMessage的id属性作为版本号,请求Config Service判断是否配置发生了变化。ReleaseMessage设计的意图是作为配置发生变化的通知,所以对于同一个Namespace,仅需要保留其最新的ReleaseMessage记录。所以,在DatabaseMessageSender中,有后台任务不断清理旧的ReleaseMessage记录。这里先留个印象后面会一一讲到
ReleaseController#publish(appId, env, clusterName, namespaceName, releaseTitle, releaseComment, releaseBy, isEmergencyPublish)
中用ReleaseMessageKeyGenerator#(appId, cluster, namespace)
来生成Release消息,然后调用MessageSender#sendMessage(message, channel)
方法发送Release消息
public interface {
/**
* 发送Message
*
* @param message 消息
* @param channel 通道(主题)
*/
void sendMessage(String message, String channel);
}
MessageSender的子类DatabaseMessageSender:
@Component
public class DatabaseMessageSender implements MessageSender {
private static final Logger logger = LoggerFactory.getLogger(DatabaseMessageSender.class);
/**
* 清理Message队列最大容量
*/
private static final int CLEAN_QUEUE_MAX_SIZE = 100;
/**
* 清理Message队列
*/
private BlockingQueue<Long> toClean = Queues.newLinkedBlockingQueue(CLEAN_QUEUE_MAX_SIZE);
/**
* 清理Message ExecutorService
*/
private final ExecutorService cleanExecutorService;
/**
* 是否停止清理Message标识
*/
private final AtomicBoolean cleanStopped;
private final ReleaseMessageRepository releaseMessageRepository;
public DatabaseMessageSender(final ReleaseMessageRepository releaseMessageRepository) {
//创建ExecutorService对象
cleanExecutorService = Executors.newSingleThreadExecutor(ApolloThreadFactory.create("DatabaseMessageSender", true));
//设置cleanStopped为false
cleanStopped = new AtomicBoolean(false);
this.releaseMessageRepository = releaseMessageRepository;
}
@Override
@Transactional
public void sendMessage(String message, String channel) {
logger.info("Sending message {} to channel {}", message, channel);
//仅允许发送APOLLO_RELEASE_TOPIC(apollo-release)
if (!Objects.equals(channel, Topics.APOLLO_RELEASE_TOPIC)) {
logger.warn("Channel {} not supported by DatabaseMessageSender!", channel);
return;
}
Tracer.logEvent("Apollo.AdminService.ReleaseMessage", message);
Transaction transaction = Tracer.newTransaction("Apollo.AdminService", "sendMessage");
try {
//保存ReleaseMessage对象
ReleaseMessage newMessage = releaseMessageRepository.save(new ReleaseMessage(message));
//添加到清理Message队列 若队列已满,添加失败,不阻塞等待
toClean.offer(newMessage.getId());
transaction.setStatus(Transaction.SUCCESS);
} catch (Throwable ex) {
logger.error("Sending message to database failed", ex);
transaction.setStatus(ex);
throw ex;
} finally {
transaction.complete();
}
}
sendMessage()
方法中主要保存ReleaseMessage对象,然后将消息ID添加到清理Message队列
清理ReleaseMessage任务:
@Component
public class DatabaseMessageSender implements MessageSender {
@PostConstruct
private void initialize() {
cleanExecutorService.submit(() -> {
while (!cleanStopped.get() && !Thread.currentThread().isInterrupted()) {
try {
//拉取
Long rm = toClean.poll(1, TimeUnit.SECONDS);
//队列非空,处理拉取到的消息
if (rm != null) {
cleanMessage(rm);
}
//队列为空,sleep,避免空跑,占用CPU
else {
TimeUnit.SECONDS.sleep(5);
}
} catch (Throwable ex) {
Tracer.logError(ex);
}
}
});
}
private void cleanMessage(Long id) {
//查询对应的ReleaseMessage对象,避免已经删除
ReleaseMessage releaseMessage = releaseMessageRepository.findById(id).orElse(null);
if (releaseMessage == null) {
return;
}
boolean hasMore = true;
//循环删除相同消息内容(message)的老消息
while (hasMore && !Thread.currentThread().isInterrupted()) {
//拉取相同消息内容的100条老消息 按照id升序排序
//老消息的定义:比当前消息编号小,即先发送的
List<ReleaseMessage> messages = releaseMessageRepository.findFirst100ByMessageAndIdLessThanOrderByIdAsc(
releaseMessage.getMessage(), releaseMessage.getId());
//删除老消息
releaseMessageRepository.deleteAll(messages);
//若拉取不足100条,说明无老消息了
hasMore = messages.size() == 100;
messages.forEach(toRemove -> Tracer.logEvent(
String.format("ReleaseMessage.Clean.%s", toRemove.getMessage()), String.valueOf(toRemove.getId())));
}
}
ReleaseMessageListener:
com.ctrip.framework.apollo.biz.message.ReleaseMessageListener
监听器接口,代码如下:
public interface ReleaseMessageListener {
/**
* 处理ReleaseMessage
*
* @param message
* @param channel 通道(主题)
*/
void handleMessage(ReleaseMessage message, String channel);
}
ReleaseMessageListener实现子类如下图:
Config Service模块:
ReleaseMessageScanner:
com.ctrip.framework.apollo.biz.message.ReleaseMessageScanner
实现org.springframework.beans.factory.InitializingBean
接口,ReleaseMessage扫描器,被Config Service使用
public class ReleaseMessageScanner implements InitializingBean {
private static final Logger logger = LoggerFactory.getLogger(ReleaseMessageScanner.class);
@Autowired
private BizConfig bizConfig;
@Autowired
private ReleaseMessageRepository releaseMessageRepository;
/**
* 从DB中扫描ReleaseMessage表的频率(毫秒)
*/
private int databaseScanInterval;
/**
* 监听器数组
*/
private List<ReleaseMessageListener> listeners;
/**
* 定时任务服务
*/
private ScheduledExecutorService executorService;
/**
* 最后扫描到的ReleaseMessage的编号
*/
private long maxIdScanned;
public ReleaseMessageScanner() {
//创建监听器数组
listeners = Lists.newCopyOnWriteArrayList();
//创建ScheduledExecutorService对象
executorService = Executors.newScheduledThreadPool(1, ApolloThreadFactory
.create("ReleaseMessageScanner", true));
}
/**
* 通过Spring调用,初始化Scan任务
*
* @throws Exception
*/
@Override
public void afterPropertiesSet() throws Exception {
databaseScanInterval = bizConfig.releaseMessageScanIntervalInMilli();
//获得最大的ReleaseMessage的编号
maxIdScanned = loadLargestMessageId();
//创建从DB中扫描ReleaseMessage表的定时任务
executorService.scheduleWithFixedDelay(() -> {
Transaction transaction = Tracer.newTransaction("Apollo.ReleaseMessageScanner", "scanMessage");
try {
//从DB中扫描ReleaseMessage
scanMessages();
transaction.setStatus(Transaction.SUCCESS);
} catch (Throwable ex) {
transaction.setStatus(ex);
logger.error("Scan and send message failed", ex);
} finally {
transaction.complete();
}
}, databaseScanInterval, databaseScanInterval, TimeUnit.MILLISECONDS);
}
/**
* 获得最大的ReleaseMessage的编号
*
* @return
*/
private long loadLargestMessageId() {
ReleaseMessage releaseMessage = releaseMessageRepository.findTopByOrderByIdDesc();
return releaseMessage == null ? 0 : releaseMessage.getId();
}
ReleaseMessageScanner中的监听器数组。通过addMessageListener()
方法,注册ReleaseMessageListener。在MessageScannerConfiguration中,调用该方法,初始化ReleaseMessageScanner的监听器们
@Configuration
public class ConfigServiceAutoConfiguration {
@Configuration
static class MessageScannerConfiguration {
private final NotificationController notificationController;
private final ConfigFileController configFileController;
private final NotificationControllerV2 notificationControllerV2;
private final GrayReleaseRulesHolder grayReleaseRulesHolder;
private final ReleaseMessageServiceWithCache releaseMessageServiceWithCache;
private final ConfigService configService;
public MessageScannerConfiguration(
final NotificationController notificationController,
final ConfigFileController configFileController,
final NotificationControllerV2 notificationControllerV2,
final GrayReleaseRulesHolder grayReleaseRulesHolder,
final ReleaseMessageServiceWithCache releaseMessageServiceWithCache,
final ConfigService configService) {
this.notificationController = notificationController;
this.configFileController = configFileController;
this.notificationControllerV2 = notificationControllerV2;
this.grayReleaseRulesHolder = grayReleaseRulesHolder;
this.releaseMessageServiceWithCache = releaseMessageServiceWithCache;
this.configService = configService;
}
@Bean
public ReleaseMessageScanner releaseMessageScanner() {
ReleaseMessageScanner releaseMessageScanner = new ReleaseMessageScanner();
//0. handle release message cache
releaseMessageScanner.addMessageListener(releaseMessageServiceWithCache);
//1. handle gray release rule
releaseMessageScanner.addMessageListener(grayReleaseRulesHolder);
//2. handle server cache
releaseMessageScanner.addMessageListener(configService);
releaseMessageScanner.addMessageListener(configFileController);
//3. notify clients
releaseMessageScanner.addMessageListener(notificationControllerV2);
releaseMessageScanner.addMessageListener(notificationController);
return releaseMessageScanner;
}
}
scanMessages()
从DB中扫描ReleaseMessage,代码如下:
public class ReleaseMessageScanner implements InitializingBean {
/**
* 从DB中扫描ReleaseMessage
*
*/
private void scanMessages() {
boolean hasMoreMessages = true;
//循环扫描消息,直到没有新的ReleaseMessage为止
while (hasMoreMessages && !Thread.currentThread().isInterrupted()) {
hasMoreMessages = scanAndSendMessages();
}
}
private boolean scanAndSendMessages() {
//current batch is 500
//获得大于maxIdScanned的500条ReleaseMessage记录,按照id升序
List<ReleaseMessage> releaseMessages =
releaseMessageRepository.findFirst500ByIdGreaterThanOrderByIdAsc(maxIdScanned);
if (CollectionUtils.isEmpty(releaseMessages)) {
return false;
}
//触发监听器
fireMessageScanned(releaseMessages);
//获得新的maxIdScanned,取最后一条记录
int messageScanned = releaseMessages.size();
maxIdScanned = releaseMessages.get(messageScanned - 1).getId();
//若拉取不足500条,说明无新消息了
return messageScanned == 500;
}
/**
* 触发监听器,处理ReleaseMessage
*
* @param messages
*/
private void fireMessageScanned(List<ReleaseMessage> messages) {
for (ReleaseMessage message : messages) {
for (ReleaseMessageListener listener : listeners) {
try {
//触发监听器
listener.handleMessage(message, Topics.APOLLO_RELEASE_TOPIC);
} catch (Throwable ex) {
Tracer.logError(ex);
logger.error("Failed to invoke message listener {}", listener.getClass(), ex);
}
}
}
}
fireMessageScanned()
方法触发ReleaseMessageListener监听器,处理ReleaseMessage。ReleaseMessageListener的子类NotificationControllerV2得到配置发布的appId+cluster+namespace后,会通知对应的客户端
3)、Config Service通知客户端
Config Service通知客户端的实现方式:
- 客户端会发起一个Http请求到Config Service的
notifications/v2
接口,也就是NotificationControllerV2,参见RemoteConfigLongPollService - NotificationControllerV2不会立即返回结果,而是通过Spring DeferredResult把请求挂起
- 如果在60秒内没有该客户端关心的配置发布,那么会返回Http状态码304给客户端
- 如果有该客户端关心的配置发布,NotificationControllerV2会调用DeferredResult的
setResult()
方法,传入有配置变化的namespace信息,同时该请求会立即返回。客户端从返回的结果中获取到配置变化的namespace后,会立即请求Config Service获取该namespace的最新配置
Admin Service发送ReleaseMessage(异步)中我们讲到NotificationControllerV2是ReleaseMessageListener的子类,ReleaseMessageScanner会扫描ReleaseMessage消息,如果有新消息会通知触发ReleaseMessageListener监听器,当然也包括NotificationControllerV2,NotificationControllerV2得到配置发布的appId+cluster+namespace后会通知对应的客户端,我们先从NotificationControllerV2看起:
在目前Apollo的实现里,如下的名词是等价的:
- 通知编号 =
ReleaseMessage.id
- Watch Key =
ReleaseMessage.message
NotificationControllerV2:
NotificationControllerV2使用DeferredResult进行异步请求处理
当一个请求到达API接口,如果该API接口的return返回值是DeferredResult,在没有超时或者DeferredResult对象设置setResult时,接口不会返回,但是Servlet容器线程会结束,DeferredResult另起线程来进行结果处理(即这种操作提升了服务短时间的吞吐能力)并setResult,如此以来这个请求不会占用服务连接池太久,如果超时或设置setResult,接口会立即返回
使用DeferredResult的流程:
浏览器发起异步请求
请求到达服务端被挂起
向浏览器进行响应,分为两种情况:
1)调用
DeferredResult.setResult()
,请求被唤醒,返回结果2)超时,返回一个你设定的结果
浏览得到响应,处理此次响应结果
关于DeferredResult的知识推荐阅读:
com.ctrip.framework.apollo.configservice.controller.NotificationControllerV2
实现ReleaseMessageListener接口,通知Controller,仅提供notifications/v2
接口
@RestController
@RequestMapping("/notifications/v2")
public class NotificationControllerV2 implements ReleaseMessageListener {
private static final Logger logger = LoggerFactory.getLogger(NotificationControllerV2.class);
/**
* Watch Key与DeferredResultWrapper的Multimap
* Watch Key等价于ReleaseMessage的通知内容message字段,appId+cluster+namespace
*/
private final Multimap<String, DeferredResultWrapper> deferredResults =
Multimaps.synchronizedSetMultimap(TreeMultimap.create(String.CASE_INSENSITIVE_ORDER, Ordering.natural()));
private static final Splitter STRING_SPLITTER =
Splitter.on(ConfigConsts.CLUSTER_NAMESPACE_SEPARATOR).omitEmptyStrings();
private static final Type notificationsTypeReference =
new TypeToken<List<ApolloConfigNotification>>() {
}.getType();
private final ExecutorService largeNotificationBatchExecutorService;
private final WatchKeysUtil watchKeysUtil;
private final ReleaseMessageServiceWithCache releaseMessageService;
private final EntityManagerUtil entityManagerUtil;
private final NamespaceUtil namespaceUtil;
private final Gson gson;
private final BizConfig bizConfig;
@Autowired
public NotificationControllerV2(
final WatchKeysUtil watchKeysUtil,
final ReleaseMessageServiceWithCache releaseMessageService,
final EntityManagerUtil entityManagerUtil,
final NamespaceUtil namespaceUtil,
final Gson gson,
final BizConfig bizConfig) {
largeNotificationBatchExecutorService = Executors.newSingleThreadExecutor(ApolloThreadFactory.create
("NotificationControllerV2", true));
this.watchKeysUtil = watchKeysUtil;
this.releaseMessageService = releaseMessageService;
this.entityManagerUtil = entityManagerUtil;
this.namespaceUtil = namespaceUtil;
this.gson = gson;
this.bizConfig = bizConfig;
}
@GetMapping
public DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> pollNotification(
@RequestParam(value = "appId") String appId,
@RequestParam(value = "cluster") String cluster,
@RequestParam(value = "notifications") String notificationsAsString,
@RequestParam(value = "dataCenter", required = false) String dataCenter,
@RequestParam(value = "ip", required = false) String clientIp) {
//1)解析notificationsAsString参数,创建ApolloConfigNotification数组
List<ApolloConfigNotification> notifications = null;
try {
notifications =
gson.fromJson(notificationsAsString, notificationsTypeReference);
} catch (Throwable ex) {
Tracer.logError(ex);
}
if (CollectionUtils.isEmpty(notifications)) {
throw new BadRequestException("Invalid format of notifications: " + notificationsAsString);
}
//过滤并创建ApolloConfigNotification Map
Map<String, ApolloConfigNotification> filteredNotifications = filterNotifications(appId, notifications);
if (CollectionUtils.isEmpty(filteredNotifications)) {
throw new BadRequestException("Invalid format of notifications: " + notificationsAsString);
}
//创建DeferredResultWrapper对象 超时时间60s
DeferredResultWrapper deferredResultWrapper = new DeferredResultWrapper(bizConfig.longPollingTimeoutInMilli());
//Namespace集合
Set<String> namespaces = Sets.newHashSetWithExpectedSize(filteredNotifications.size());
//客户端的通知Map key为namespace名,value为通知编号(ReleaseMessage.id)
Map<String, Long> clientSideNotifications = Maps.newHashMapWithExpectedSize(filteredNotifications.size());
for (Map.Entry<String, ApolloConfigNotification> notificationEntry : filteredNotifications.entrySet()) {
String normalizedNamespace = notificationEntry.getKey();
ApolloConfigNotification notification = notificationEntry.getValue();
//添加到namespaces中
namespaces.add(normalizedNamespace);
//添加到clientSideNotifications中
clientSideNotifications.put(normalizedNamespace, notification.getNotificationId());
if (!Objects.equals(notification.getNamespaceName(), normalizedNamespace)) {
deferredResultWrapper.recordNamespaceNameNormalizedResult(notification.getNamespaceName(), normalizedNamespace);
}
}
//组装Watch Key Multimap
Multimap<String, String> watchedKeysMap =
watchKeysUtil.assembleAllWatchKeys(appId, cluster, namespaces, dataCenter);
//生成Watch Key集合
Set<String> watchedKeys = Sets.newHashSet(watchedKeysMap.values());
/**
* 1、set deferredResult before the check, for avoid more waiting
* If the check before setting deferredResult,it may receive a notification the next time
* when method handleMessage is executed between check and set deferredResult.
*/
//注册超时事件
deferredResultWrapper
.onTimeout(() -> logWatchedKeys(watchedKeys, "Apollo.LongPoll.TimeOutKeys"));
//注册结束事件
deferredResultWrapper.onCompletion(() -> {
//unregister all keys
//移除Watch Key+DeferredResultWrapper出deferredResults
for (String key : watchedKeys) {
deferredResults.remove(key, deferredResultWrapper);
}
logWatchedKeys(watchedKeys, "Apollo.LongPoll.CompletedKeys");
});
//register all keys
//注册Watch Key+DeferredResultWrapper到deferredResults中,等待配置发生变化后通知
for (String key : watchedKeys) {
this.deferredResults.put(key, deferredResultWrapper);
}
logWatchedKeys(watchedKeys, "Apollo.LongPoll.RegisteredKeys");
logger.debug("Listening {} from appId: {}, cluster: {}, namespace: {}, datacenter: {}",
watchedKeys, appId, cluster, namespaces, dataCenter);
//2)获得Watch Key集合中,每个Watch Key对应的ReleaseMessage记录
List<ReleaseMessage> latestReleaseMessages =
releaseMessageService.findLatestReleaseMessagesGroupByMessages(watchedKeys);
/**
* 手动关闭EntityManager
* 因为对于async请求,Spring在请求完成之前不会这样做
* 这是不可接受的,因为我们正在做长轮询——意味着db连接将被保留很长时间
* 实际上,下面的过程,我们已经不需要db连接,因此进行关闭
*/
entityManagerUtil.closeEntityManager();
//3)获得新的ApolloConfigNotification通知数组
List<ApolloConfigNotification> newNotifications =
getApolloConfigNotifications(namespaces, clientSideNotifications, watchedKeysMap,
latestReleaseMessages);
//若有新的通知,直接设置结果
if (!CollectionUtils.isEmpty(newNotifications)) {
deferredResultWrapper.setResult(newNotifications);
}
return deferredResultWrapper.getResult();
}
/**
* 通过ReleaseMessage的消息内容,获得对应namespace的名字 拆分appId+cluster+namespace拿到namespace
*/
private static final Function<String, String> retrieveNamespaceFromReleaseMessage =
releaseMessage -> {
if (Strings.isNullOrEmpty(releaseMessage)) {
return null;
}
List<String> keys = STRING_SPLITTER.splitToList(releaseMessage);
//message should be appId+cluster+namespace
if (keys.size() != 3) {
logger.error("message format invalid - {}", releaseMessage);
return null;
}
return keys.get(2);
};
pollNotification()
核心源码如下:
-
代码1)解析notificationsAsString参数,解析成
List<ApolloConfigNotification>
因为一个客户端可以订阅多个Namespace,所以该参数是List。该接口返回的结果也是
List<ApolloConfigNotification>
,仅返回配置发生变化的Namespace对应的ApolloConfigNotification。也就说,当有几个配置发生变化的 Namespace ,返回几个对应的 ApolloConfigNotification 。另外,客户端接收到返回后,会增量合并到本地的配置通知信息。客户端下次请求时,使用合并后的配置通知信息。这里先留个印象后面解析Apollo Client源码的时候会讲到ApolloConfigNotification源码如下:
public class ApolloConfigNotification { /** * namespace名字 */ private String namespaceName; /** * 最新通知编号 * 目前使用ReleaseMessage.id */ private long notificationId; /** * 通知消息集合 */ private volatile ApolloNotificationMessages messages;
public class ApolloNotificationMessages { /** * 明细Map * key:appId+clusterName+namespace,例如:100004458+default+application * value:通知编号 */ private Map<String, Long> details;
-
代码1)和代码2)之间是有关于DeferredResultWrapper的操作:
com.ctrip.framework.apollo.configservice.wrapper.DeferredResultWrapper
是DeferredResult包装器,封装DeferredResult的公用方法public class DeferredResultWrapper implements Comparable<DeferredResultWrapper> { /** * 未修改时的ResponseEntity响应,使用302状态码 */ private static final ResponseEntity<List<ApolloConfigNotification>> NOT_MODIFIED_RESPONSE_LIST = new ResponseEntity<>(HttpStatus.NOT_MODIFIED); /** * 归一化和原始的Namespace的名字的Map */ private Map<String, String> normalizedNamespaceNameToOriginalNamespaceName; /** * 响应的DeferredResult对象 */ private DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> result; public DeferredResultWrapper(long timeoutInMilli) { result = new DeferredResult<>(timeoutInMilli, NOT_MODIFIED_RESPONSE_LIST); } public void onTimeout(Runnable timeoutCallback) { result.onTimeout(timeoutCallback); } public void onCompletion(Runnable completionCallback) { result.onCompletion(completionCallback); } public void setResult(ApolloConfigNotification notification) { setResult(Lists.newArrayList(notification)); } /** * The namespace name is used as a key in client side, so we have to return the original one instead of the correct one */ public void setResult(List<ApolloConfigNotification> notifications) { if (normalizedNamespaceNameToOriginalNamespaceName != null) { //恢复被归一化的Namespace的名字为原始的Namespace的名字 notifications.stream().filter(notification -> normalizedNamespaceNameToOriginalNamespaceName.containsKey (notification.getNamespaceName())).forEach(notification -> notification.setNamespaceName( normalizedNamespaceNameToOriginalNamespaceName.get(notification.getNamespaceName()))); } //设置结果,并使用200状态码 result.setResult(new ResponseEntity<>(notifications, HttpStatus.OK)); } public DeferredResult<ResponseEntity<List<ApolloConfigNotification>>> getResult() { return result; }
主要是创建DeferredResultWrapper对象(超时时间60s);拿到客户端的通知Map,key为namespace名,value为客户端通知编号ReleaseMessage.id,如果Config Service的ReleaseMessage.id大于客户端本地的ReleaseMessage.id,则对应Watch Key(Watch Key = ReleaseMessage.message = appId+cluster+namespace)有配置更新;拿到Watch Key的集合;注册Watch Key+DeferredResultWrapper到deferredResults中;等待配置发生变化后通知注册超时事件和结束事件(从deferredResults中移除Watch Key+DeferredResultWrapper)
-
代码2)获得Watch Key集合中,每个Watch Key对应的ReleaseMessage记录,这里是从Config Service本地Cache中获取的
-
代码3)调用
getApolloConfigNotifications()
方法获得新的ApolloConfigNotification通知数组,如果此时就有新的通知,直接设置结果,getApolloConfigNotifications()
源码如下:@RestController @RequestMapping("/notifications/v2") public class NotificationControllerV2 implements ReleaseMessageListener { /** * 获得新的ApolloConfigNotification通知数组 * * @param namespaces * @param clientSideNotifications * @param watchedKeysMap * @param latestReleaseMessages * @return */ private List<ApolloConfigNotification> getApolloConfigNotifications(Set<String> namespaces, Map<String, Long> clientSideNotifications, Multimap<String, String> watchedKeysMap, List<ReleaseMessage> latestReleaseMessages) { //创建ApolloConfigNotification数组 List<ApolloConfigNotification> newNotifications = Lists.newArrayList(); if (!CollectionUtils.isEmpty(latestReleaseMessages)) { //创建最新通知的Map 其中Key为Watch Key Map<String, Long> latestNotifications = Maps.newHashMap(); for (ReleaseMessage releaseMessage : latestReleaseMessages) { latestNotifications.put(releaseMessage.getMessage(), releaseMessage.getId()); } //循环Namespace的名字的集合,判断是否有配置更新 for (String namespace : namespaces) { long clientSideId = clientSideNotifications.get(namespace); long latestId = ConfigConsts.NOTIFICATION_ID_PLACEHOLDER; //获得Namespace对应的Watch Key集合 Collection<String> namespaceWatchedKeys = watchedKeysMap.get(namespace); //获得最大的通知编号 for (String namespaceWatchedKey : namespaceWatchedKeys) { long namespaceNotificationId = latestNotifications.getOrDefault(namespaceWatchedKey, ConfigConsts.NOTIFICATION_ID_PLACEHOLDER); if (namespaceNotificationId > latestId) { latestId = namespaceNotificationId; } } //若服务器的通知编号大于客户端的通知编号,意味着有配置更新 if (latestId > clientSideId) { //创建ApolloConfigNotification对象 ApolloConfigNotification notification = new ApolloConfigNotification(namespace, latestId); //循环添加通知编号到ApolloConfigNotification中 namespaceWatchedKeys.stream().filter(latestNotifications::containsKey).forEach(namespaceWatchedKey -> notification.addMessage(namespaceWatchedKey, latestNotifications.get(namespaceWatchedKey))); //添加ApolloConfigNotification对象到结果 newNotifications.add(notification); } } } return newNotifications; }
其实核心逻辑就是若服务器的通知编号大于客户端的通知编号,意味着有配置更新
-
Admin Service发送ReleaseMessage(异步)中我们说到,ReleaseMessageScanner中的
fireMessageScanned()
方法触发ReleaseMessageListener监听器,处理ReleaseMessage。ReleaseMessageListener的子类NotificationControllerV2得到配置发布的appId+cluster+namespace后,会通知对应的客户端,NotificationControllerV2对应实现源码如下:
@RestController
@RequestMapping("/notifications/v2")
public class NotificationControllerV2 implements ReleaseMessageListener {
@Override
public void handleMessage(ReleaseMessage message, String channel) {
logger.info("message received - channel: {}, message: {}", channel, message);
String content = message.getMessage();
Tracer.logEvent("Apollo.LongPoll.Messages", content);
//仅处理APOLLO_RELEASE_TOPIC
if (!Topics.APOLLO_RELEASE_TOPIC.equals(channel) || Strings.isNullOrEmpty(content)) {
return;
}
//获得对应的Namespace的名字
String changedNamespace = retrieveNamespaceFromReleaseMessage.apply(content);
if (Strings.isNullOrEmpty(changedNamespace)) {
logger.error("message format invalid - {}", content);
return;
}
//deferredResults存在对应的Watch Key
if (!deferredResults.containsKey(content)) {
return;
}
//create a new list to avoid ConcurrentModificationException
//创建DeferredResultWrapper数组,避免并发问题
List<DeferredResultWrapper> results = Lists.newArrayList(deferredResults.get(content));
//创建ApolloConfigNotification对象
ApolloConfigNotification configNotification = new ApolloConfigNotification(changedNamespace, message.getId());
configNotification.addMessage(content, message.getId());
//do async notification if too many clients
//1)若需要通知的客户端过多,使用ExecutorService异步通知
if (results.size() > bizConfig.releaseMessageNotificationBatch()) {
largeNotificationBatchExecutorService.submit(() -> {
logger.debug("Async notify {} clients for key {} with batch {}", results.size(), content,
bizConfig.releaseMessageNotificationBatch());
for (int i = 0; i < results.size(); i++) {
//每N个客户端,sleep一段时间
if (i > 0 && i % bizConfig.releaseMessageNotificationBatch() == 0) {
try {
TimeUnit.MILLISECONDS.sleep(bizConfig.releaseMessageNotificationBatchIntervalInMilli());
} catch (InterruptedException e) {
//ignore
}
}
logger.debug("Async notify {}", results.get(i));
//设置结果
results.get(i).setResult(configNotification);
}
});
return;
}
logger.debug("Notify {} clients for key {}", results.size(), content);
for (DeferredResultWrapper result : results) {
//2)设置结果
result.setResult(configNotification);
}
logger.debug("Notification completed");
}
-
【异步】处理:
当需要通知的客户端过多,使用ExecutorService异步通知,避免惊群效应
假设一个公共Namespace有10W台机器使用,如果该公共Namespace发布时直接下发配置更新消息的话,就会导致这10W台机器一下子都来请求配置,这动静就有点大了,而且对Config Service的压力也会比较大
默认超过100个客户端sleep 100毫秒
-
【同步】处理:
代码2)循环调用DeferredResultWrapper的setResult()
方法,设置DeferredResult的结果,从而结束长轮询
参考:
https://ctripcorp.github.io/apollo/#/zh/design/apollo-design
https://www.iocoder.cn/Apollo
https://time.geekbang.org/column/article/175164