Nacos源码分析02-服务端服务注册

Nacos源码分析02-服务端服务注册

本系列博客，采用官方源码版本为2.0.3

1. Client&gRPC长连接

Nacos2.0 新服务模型

  补充一下，在Nacos2.0之后，新支持了gRPC长连接。一个客户端gRPC长连接对应一个Client，每个Client有自己唯一的id（clientId）。 Client负责管理一个客户端的服务实例注册Publish和服务订阅Subscribe。

public interface Client {
    // 客户端id/gRPC的connectionId
    String getClientId();

    // 是否临时客户端
    boolean isEphemeral();
    // 客户端更新时间
    void setLastUpdatedTime();
    long getLastUpdatedTime();

    // 服务实例注册/注销/查询
    boolean addServiceInstance(Service service, InstancePublishInfo instancePublishInfo);
    InstancePublishInfo removeServiceInstance(Service service);
    InstancePublishInfo getInstancePublishInfo(Service service);
    Collection<Service> getAllPublishedService();

    // 服务订阅/取消订阅/查询订阅
    boolean addServiceSubscriber(Service service, Subscriber subscriber);
    boolean removeServiceSubscriber(Service service);
    Subscriber getSubscriber(Service service);
    Collection<Service> getAllSubscribeService();
    
    // 生成同步给其他节点的client数据
    ClientSyncData generateSyncData();
    
    // 是否过期
    boolean isExpire(long currentTime);
    
    // 释放资源
    void release();
}

2. 请求处理

2.1. 基于http的流程

2.1.1. InstanceController#registerInstance

  通过RESTful API的路径可以找到，HTTP协议会调取InstanceController下的register(HttpServletRequest)方法。在这个方法中，会把请求中的数据进行解析，还原成Instance。

  getInstanceOperator()，就是判断是否支持Grpc协议，由于使用2.0.3版本，所以此处比如选择instanceServiceV2。instanceServiceV2的类型为InstanceOperatorClientImpl，instanceServiceV1的类型为InstanceOperatorServiceImpl。

  综上所述,最终实际调用了InstanceOperatorClientImpl#registerInstance()方法。

@CanDistro
@PostMapping
@Secured(parser = NamingResourceParser.class, action = ActionTypes.WRITE)
public String register(HttpServletRequest request) throws Exception {

    final String namespaceId = WebUtils
        .optional(request, CommonParams.NAMESPACE_ID, Constants.DEFAULT_NAMESPACE_ID);
    final String serviceName = WebUtils.required(request, CommonParams.SERVICE_NAME);
    NamingUtils.checkServiceNameFormat(serviceName);

    final Instance instance = HttpRequestInstanceBuilder.newBuilder()
        .setDefaultInstanceEphemeral(switchDomain.isDefaultInstanceEphemeral()).setRequest(request).build();

    getInstanceOperator().registerInstance(namespaceId, serviceName, instance);
    return "ok";
}

private InstanceOperator getInstanceOperator() {
    return upgradeJudgement.isUseGrpcFeatures() ? instanceServiceV2 : instanceServiceV1;
}

2.1.2. InstanceOperatorClientImpl.registerInstance

  通过构造器可以看出clientOperationService的实现类是ClientOperationServiceProxy。

@Override
public void registerInstance(String namespaceId, String serviceName, Instance instance) {
    //判断是否为瞬时对象（临时客户端）
    boolean ephemeral = instance.isEphemeral();
    //获取客户端ID
    String clientId = IpPortBasedClient.getClientId(instance.toInetAddr(), ephemeral);
    //通过客户端ID创建客户端连接
    createIpPortClientIfAbsent(clientId);
    //拼装服务
    Service service = getService(namespaceId, serviceName, ephemeral);
    //具体注册服务
    clientOperationService.registerInstance(service, instance, clientId);
}

public InstanceOperatorClientImpl(ClientManagerDelegate clientManager,
        ClientOperationServiceProxy clientOperationService, ServiceStorage serviceStorage,
        NamingMetadataOperateService metadataOperateService, NamingMetadataManager metadataManager,
        SwitchDomain switchDomain, UdpPushService pushService) {
    this.clientManager = clientManager;
    this.clientOperationService = clientOperationService;
    this.serviceStorage = serviceStorage;
    this.metadataOperateService = metadataOperateService;
    this.metadataManager = metadataManager;
    this.switchDomain = switchDomain;
    this.pushService = pushService;
}

2.1.3. ClientOperationServiceProxy#registerInstance

  这种Proxy的类，在客户端注册发现那边也出现过，此处就是判断instance是否是临时的来选择具体的实现类。

• ephemeralClientOperationService的类型为EphemeralClientOperationServiceImpl
• persistentClientOperationService的类型为PersistentClientOperationServiceImpl

@Override
public void registerInstance(Service service, Instance instance, String clientId) {
    final ClientOperationService operationService = chooseClientOperationService(instance);
    operationService.registerInstance(service, instance, clientId);
}
    
private ClientOperationService chooseClientOperationService(final Instance instance) {
    return instance.isEphemeral() ? ephemeralClientOperationService : persistentClientOperationService;
}

2.2. 基于gRPC的流程

2.2.1. GrpcRequestAcceptor#request

  GrpcRequestAcceptor#request核心就是收到请求后，基于gRPC协议解析请求并转成Request对象。然后通过connection拼装出请求元数据，最后调用requestHandler处理器的处理函数handleRequest(request, requestMeta)。requestHandler的实际类型为InstanceRequestHandler。

@Override
public void request(Payload grpcRequest, StreamObserver<Payload> responseObserver) {
    //此处省略一堆代码
    
    Request request = (Request) parseObj;
    try {
        Connection connection = connectionManager.getConnection(CONTEXT_KEY_CONN_ID.get());
        RequestMeta requestMeta = new RequestMeta();
        requestMeta.setClientIp(connection.getMetaInfo().getClientIp());
        requestMeta.setConnectionId(CONTEXT_KEY_CONN_ID.get());
        requestMeta.setClientVersion(connection.getMetaInfo().getVersion());
        requestMeta.setLabels(connection.getMetaInfo().getLabels());
        connectionManager.refreshActiveTime(requestMeta.getConnectionId());
        // 处理请求
        Response response = requestHandler.handleRequest(request, requestMeta);
        Payload payloadResponse = GrpcUtils.convert(response);
        traceIfNecessary(payloadResponse, false);
        responseObserver.onNext(payloadResponse);
        responseObserver.onCompleted();
    } catch (Throwable e) {
        Loggers.REMOTE_DIGEST
                .error("[{}] Fail to handle request from connection [{}] ,error message :{}", "grpc", connectionId,
                        e);
        Payload payloadResponse = GrpcUtils.convert(buildErrorResponse(
                (e instanceof NacosException) ? ((NacosException) e).getErrCode() : ResponseCode.FAIL.getCode(),
                e.getMessage()));
        traceIfNecessary(payloadResponse, false);
        responseObserver.onNext(payloadResponse);
        responseObserver.onCompleted();
    }
}

2.2.2. InstanceRequestHandler.handleRequest

  InstanceRequestHandler中的handleRequest方法继承自RequestHandler，RequestHandler的handleRequest方法核心逻辑是调用过滤器Filter，然后在调用抽象方法handle(request, meta)。

  InstanceRequestHandler中的handle方法就很简单了，基于请求参数拼装Service，然后基于请求类型处理请求。

  拼装Service的时候可以发现Service.newService()中的ephemeral属性为固定值true，所以2.0.3版本的nacos，基于grpc只能注册临时服务。同时InstanceRequestHandler中的clientOperationService属性也选择了EphemeralClientOperationServiceImpl为实现。

public Response handleRequest(T request, RequestMeta meta) throws NacosException {
    for (AbstractRequestFilter filter : requestFilters.filters) {
        try {
            Response filterResult = filter.filter(request, meta, this.getClass());
            if (filterResult != null && !filterResult.isSuccess()) {
                return filterResult;
            }
        } catch (Throwable throwable) {
            Loggers.REMOTE.error("filter error", throwable);
        }
        
    }
    return handle(request, meta);
}

@Override
@Secured(action = ActionTypes.WRITE, parser = NamingResourceParser.class)
public InstanceResponse handle(InstanceRequest request, RequestMeta meta) throws NacosException {
    Service service = Service
            .newService(request.getNamespace(), request.getGroupName(), request.getServiceName(), true);
    switch (request.getType()) {
        case NamingRemoteConstants.REGISTER_INSTANCE:
            return registerInstance(service, request, meta);
        case NamingRemoteConstants.DE_REGISTER_INSTANCE:
            return deregisterInstance(service, request, meta);
        default:
            throw new NacosException(NacosException.INVALID_PARAM,
                    String.format("Unsupported request type %s", request.getType()));
    }
}

private InstanceResponse registerInstance(Service service, InstanceRequest request, RequestMeta meta) {
    clientOperationService.registerInstance(service, request.getInstance(), meta.getConnectionId());
    return new InstanceResponse(NamingRemoteConstants.REGISTER_INSTANCE);
}

3. 服务注册

  通过阅读不同的协议处理代理可以发现，ClientOperationService#registerInstance才是服务注册的核心代码入口。ClientOperationService有三个实现类，EphemeralClientOperationServiceImpl、PersistentClientOperationServiceImpl和ClientOperationServiceProxy。

ClientOperationService

3.1. EphemeralClientOperationServiceImpl.registerInstance

  EphemeralClientOperationServiceImpl是ClientOperationService的实现类之一，其作用就是管理临时实例。

 @Override
public void registerInstance(Service service, Instance instance, String clientId) {
    //确保Service单例存在
    Service singleton = ServiceManager.getInstance().getSingleton(service);
    
    //根据客户端id，找到客户端
    Client client = clientManager.getClient(clientId);
    if (!clientIsLegal(client, clientId)) {
        return;
    }
    //客户端Instance模型，转换为服务端Instance模型
    InstancePublishInfo instanceInfo = getPublishInfo(instance);
    //将Instance储存到Client里
    client.addServiceInstance(singleton, instanceInfo);
    client.setLastUpdatedTime();
    
    //建立Service与ClientId的关系
    NotifyCenter.publishEvent(new ClientOperationEvent.ClientRegisterServiceEvent(singleton, clientId));
    NotifyCenter
        .publishEvent(new MetadataEvent.InstanceMetadataEvent(singleton, instanceInfo.getMetadataId(), false));
}

3.1.1. ServiceManager.getInstance().getSingleton(service)

Service singleton = ServiceManager.getInstance().getSingleton(service);

  通过从getSingleton()可以看出，当调用这个注册方法的时候ServiceManager负责管理Service单例。

public class ServiceManager {
    
    private static final ServiceManager INSTANCE = new ServiceManager();
    
    private final ConcurrentHashMap<Service, Service> singletonRepository;
    
    private final ConcurrentHashMap<String, Set<Service>> namespaceSingletonMaps;

    public Service getSingleton(Service service) {
        singletonRepository.putIfAbsent(service, service);
        Service result = singletonRepository.get(service);
        
        namespaceSingletonMaps.computeIfAbsent(result.getNamespace(), (namespace) -> new ConcurrentHashSet<>());
        namespaceSingletonMaps.get(result.getNamespace()).add(result);
        
        return result;
    }
}

public class Service{
    
    @Override
    public boolean equals(Object o) {
        if (this == o) {
            return true;
        }
        if (!(o instanceof Service)) {
            return false;
        }
        Service service = (Service) o;
        return namespace.equals(service.namespace) && group.equals(service.group) && name.equals(service.name);
    }
}

3.1.2. clientManager.getClient

Client client = clientManager.getClient(clientId);

  此处clientManager对象的实现类是ClientManagerDelegate，ClientManagerDelegate内部通过clientId来判断选择哪个实现类。通过NamingTest用例，此处获得了connectionBasedClientManager对象。

@Override
public Client getClient(String clientId) {
    return getClientManagerById(clientId).getClient(clientId);
}
    
private ClientManager getClientManagerById(String clientId) {
    if (isConnectionBasedClient(clientId)) {
        return connectionBasedClientManager;
    }
    return clientId.endsWith(SUFFIX) ? persistentIpPortClientManager : ephemeralIpPortClientManager;
}

  ConnectionBasedClientManager实现类负责管理长连接clientId与Client模型的映射关系。

private final ConcurrentMap<String, ConnectionBasedClient> clients = new ConcurrentHashMap<>();
    
// 根据clientId查询Client
public Client getClient(String clientId) {
    return clients.get(clientId);
}

3.1.3. client.addServiceInstance

client.addServiceInstance(singleton, instanceInfo);

  ConnectionBasedClientManager中将返回ConnectionBasedClient类型，而ConnectionBasedClient继承自AbstractClient。这决定了client.addServiceInstance最终执行了AbstractClient的方法。

  AbstractClient负责存储当前客户端的服务注册表，即Service与Instance的关系。注意对于单个客户端来说，同一个服务只能注册一个实例。

protected final ConcurrentHashMap<Service, InstancePublishInfo> publishers = new ConcurrentHashMap<>(16, 0.75f, 1);
   
@Override
public boolean addServiceInstance(Service service, InstancePublishInfo instancePublishInfo) {
    if (null == publishers.put(service, instancePublishInfo)) {
        //监控指标自增实例数
        MetricsMonitor.incrementInstanceCount();
    }
    NotifyCenter.publishEvent(new ClientEvent.ClientChangedEvent(this));
    Loggers.SRV_LOG.info("Client change for service {}, {}", service, getClientId());
    return true;
}

3.1.4. ClientOperationEvent.ClientRegisterServiceEvent

NotifyCenter.publishEvent(new ClientOperationEvent.ClientRegisterServiceEvent(singleton, clientId));

  ClientOperationEvent.ClientRegisterServiceEvent事件件会被ClientServiceIndexesManager监听并处理，建立Service与发布Client的关系就是为了加速查询。

  ClientServiceIndexesManager类中维护了两组映射表，分别是【Service与发布clientId】和【Service与订阅clientId】。

  这个索引关系建立以后，还会触发ServiceEvent.ServiceChangedEvent事件，代表服务注册表变更,由ClientServiceIndexesManager监听处理，

• 通知订阅客户端，
• Nacos集群数据同步。

private final ConcurrentMap<Service, Set<String>> publisherIndexes = new ConcurrentHashMap<>();
    
private final ConcurrentMap<Service, Set<String>> subscriberIndexes = new ConcurrentHashMap<>();

private void handleClientOperation(ClientOperationEvent event) {
    Service service = event.getService();
    String clientId = event.getClientId();
    if (event instanceof ClientOperationEvent.ClientRegisterServiceEvent) {
        addPublisherIndexes(service, clientId);
    } else if (event instanceof ClientOperationEvent.ClientDeregisterServiceEvent) {
        removePublisherIndexes(service, clientId);
    } else if (event instanceof ClientOperationEvent.ClientSubscribeServiceEvent) {
        addSubscriberIndexes(service, clientId);
    } else if (event instanceof ClientOperationEvent.ClientUnsubscribeServiceEvent) {
        removeSubscriberIndexes(service, clientId);
    }
}

//建立Service与发布Client的关系
private void addPublisherIndexes(Service service, String clientId) {
    publisherIndexes.computeIfAbsent(service, (key) -> new ConcurrentHashSet<>());
    publisherIndexes.get(service).add(clientId);
    NotifyCenter.publishEvent(new ServiceEvent.ServiceChangedEvent(service, true));
}