Nacos GRPC通信模块分析

Nacos通信模块分析

Server

整体概览

在2.0版本中Nacos采用了Grpc作为通信模块，BaseRpcServer定义了基本的服务启动关闭的接口，BaseGrpcServer则实现了基本的Server模块功能，GrpcClusterServer用于集群节点之间的交互，GrpcSdkServer用于客户端与服务器的交互，分别定义了各自的执行线程池。

BaseGrpcServer

BaseGrpcServer利用Grpc实现了远程Server端点的功能，首先初始化了一个服务调用的拦截器，在这个拦截器中获取到远程调用的属性放入线程上下文中，注意的是这里的getInternalChannel是通过反射获取到了Netty Channel在grpc的一个封装。

ServerInterceptor serverInterceptor = new ServerInterceptor() {
   
    @Override
    public <T, S> ServerCall.Listener<T> interceptCall(ServerCall<T, S> call, Metadata headers,
            ServerCallHandler<T, S> next) {
   
        Context ctx = Context.current()
                .withValue(CONTEXT_KEY_CONN_ID, call.getAttributes().get(TRANS_KEY_CONN_ID))
                .withValue(CONTEXT_KEY_CONN_REMOTE_IP, call.getAttributes().get(TRANS_KEY_REMOTE_IP))
                .withValue(CONTEXT_KEY_CONN_REMOTE_PORT, call.getAttributes().get(TRANS_KEY_REMOTE_PORT))
                .withValue(CONTEXT_KEY_CONN_LOCAL_PORT, call.getAttributes().get(TRANS_KEY_LOCAL_PORT));
        if (REQUEST_BI_STREAM_SERVICE_NAME.equals(call.getMethodDescriptor().getServiceName())) {
   
            Channel internalChannel = getInternalChannel(call);
            ctx = ctx.withValue(CONTEXT_KEY_CHANNEL, internalChannel);
        }
        return Contexts.interceptCall(ctx, call, headers, next);
    }
};

接着就是初始化方法的处理，不是采用传统的定义proto的形式，而是采用手工注册的方式，支持一元调用（一次往返的请求）以及双向流式调用。其中传输的请求体与响应体为Payload类，是由protobuf定义生成而来的，利用了any类型来支持多类型数据格式。

private void addServices(MutableHandlerRegistry handlerRegistry, ServerInterceptor... serverInterceptor) {
   
    
    // unary common call register.
    //定义Method
    final MethodDescriptor<Payload, Payload> unaryPayloadMethod = MethodDescriptor.<Payload, Payload>newBuilder()
            .setType(MethodDescriptor.MethodType.UNARY)
            .setFullMethodName(MethodDescriptor.generateFullMethodName(REQUEST_SERVICE_NAME, REQUEST_METHOD_NAME))
            .setRequestMarshaller(ProtoUtils.marshaller(Payload.getDefaultInstance()))
            .setResponseMarshaller(ProtoUtils.marshaller(Payload.getDefaultInstance())).build();
    
    //定义服务处理方法回调
    final ServerCallHandler<Payload, Payload> payloadHandler = ServerCalls
            .asyncUnaryCall((request, responseObserver) -> {
   
                grpcCommonRequestAcceptor.request(request, responseObserver);
            });
    
    //定义Servie
    final ServerServiceDefinition serviceDefOfUnaryPayload = ServerServiceDefinition.builder(REQUEST_SERVICE_NAME)
            .addMethod(unaryPayloadMethod, payloadHandler).build();
    handlerRegistry.addService(ServerInterceptors.intercept(serviceDefOfUnaryPayload, serverInterceptor));
    
    // bi stream register.
    final ServerCallHandler<Payload, Payload> biStreamHandler = ServerCalls.asyncBidiStreamingCall(
            (responseObserver) -> grpcBiStreamRequestAcceptor.requestBiStream(responseObserver));
    
    final MethodDescriptor<Payload, Payload> biStreamMethod = MethodDescriptor.<Payload, Payload>newBuilder()
            .setType(MethodDescriptor.MethodType.BIDI_STREAMING).setFullMethodName(MethodDescriptor
                    .generateFullMethodName(REQUEST_BI_STREAM_SERVICE_NAME, REQUEST_BI_STREAM_METHOD_NAME))
            .setRequestMarshaller(ProtoUtils.marshaller(Payload.newBuilder().build()))
            .setResponseMarshaller(ProtoUtils.marshaller(Payload.getDefaultInstance())).build();
    
    final ServerServiceDefinition serviceDefOfBiStream = ServerServiceDefinition
            .builder(REQUEST_BI_STREAM_SERVICE_NAME).addMethod(biStreamMethod, biStreamHandler).build();
    handlerRegistry.addService(ServerInterceptors.intercept(serviceDefOfBiStream, serverInterceptor));
    
}

最后则是设置相关的Server属性以及对连接的管理，这里的transportReady会在连接建立时根据当前的连接的属性构造一个新的Attributes，主要是为每个连接建议一个connectionId，和前面提到的拦截器相呼应。并在连接断开时从connectionManager移除连接。

server = ServerBuilder.forPort(getServicePort()).executor(getRpcExecutor())
        .maxInboundMessageSize(getInboundMessageSize()).fallbackHandlerRegistry(handlerRegistry)
        .compressorRegistry(CompressorRegistry.getDefaultInstance())
        .decompressorRegistry(DecompressorRegistry.getDefaultInstance())
        .addTransportFilter(new ServerTransportFilter() {
   
            @Override
            public Attributes transportReady(Attributes transportAttrs) {
   
                InetSocketAddress remoteAddress = (InetSocketAddress) transportAttrs
                        .get(Grpc.TRANSPORT_ATTR_REMOTE_ADDR);
                InetSocketAddress localAddress = (InetSocketAddress) transportAttrs
                        .get(Grpc.TRANSPORT_ATTR_LOCAL_ADDR);
                int remotePort = remoteAddress.getPort();
                int localPort = localAddress.getPort();
                String remoteIp = remoteAddress.getAddress().getHostAddress();
                Attributes attrWrapper = transportAttrs.toBuilder()
                        .set(TRANS_KEY_CONN_ID, System.currentTimeMillis() + "_" + remoteIp + "_" + remotePort)
                        .set(TRANS_KEY_REMOTE_IP, remoteIp).set(TRANS_KEY_REMOTE_PORT, remotePort)
                        .set(TRANS_KEY_LOCAL_PORT, localPort).build();
                String connectionId = attrWrapper.get(TRANS_KEY_CONN_ID);
                Loggers.REMOTE_DIGEST.info("Connection transportReady,connectionId = {} ", connectionId);
                return attrWrapper;
                
            }
            
            @Override
            public void transportTerminated(Attributes transportAttrs) {
   
                String connectionId = null;
                try {
   
                    connectionId = transportAttrs.get(TRANS_KEY_CONN_ID);
                } catch (Exception e) {
   
                    // Ignore
                }
                if (StringUtils.isNotBlank(connectionId)) {
   
                    Loggers.REMOTE_DIGEST
                            .info("Connection transportTerminated,connectionId = {} ", connectionId);
                    connectionManager.unregister(connectionId);
                }
            }
        }).build();

ConnectionManager

ConnectionManager用于管理与Server之间建立的长连接，在Nacos中，只有建立了双向流的连接才会被纳入到管理之中，Nacos本身也是利用双向流来完成Server端的主动推送。与Sentinel/Seata类似的，都是采用ConcurrentHashMap来保存对应的Connection连接对象，在注册连接时还会检查当前的连接相关的限制，对客户端进行一个统计计数，并调用事件回调通知监听器有连接建立。注意这里使用了synchronized的关键字，也就说明回调函数实际上是串行执行的（举个例子，RpcAckCallbackInitorOrCleaner会在连接建立/断开时初始化/清理RpcAckCallbackSynchronizer中连接对应的Map Entry）

public synchronized boolean register(String connectionId, Connection connection) {
   
    
    if (connection.isConnected()) {
   
        if (connections.containsKey(connectionId)) {
   
            return true;
        }
        if (!checkLimit(connection)) {
   
            return false;
        }
        if (traced(connection.getMetaInfo().clientIp)) {
   
            connection.setTraced(true);
        }
        connections.put(connectionId, connection);
        connectionForClientIp.get(connection.getMetaInfo().clientIp).getAndIncrement();
        
        clientConnectionEventListenerRegistry.notifyClientConnected(connection);
        Loggers.REMOTE_DIGEST
                .info("new connection registered successfully, connectionId = {},connection={} ", connectionId,
                        connection);