Dubbo3.3之Triple协议客户端调用源码分析

smalltalkhyb

于 2024-10-06 22:25:53 发布

阅读量325

点赞数 12

分类专栏： dubbo triple协议文章标签： dubbo rpc

本文链接：https://blog.csdn.net/smalltalkhyb/article/details/142657378

版权

dubbo 同时被 2 个专栏收录

1 篇文章 0 订阅

订阅专栏

triple协议

1 篇文章 0 订阅

订阅专栏

Triple X 全新升级背景

在 Dubbo 的早期应用中，虽然其在数据中心内部的服务互通中展现了卓越的性能，但随着技术演进和应用场景的扩展，原有架构逐渐暴露出了一些瓶颈。这些瓶颈在跨区域、跨云的环境中尤为明显，尤其是在 Web 框架与 RPC 框架之间的频繁切换中，开发复杂性和系统性能都受到了影响。

传统架构的痛点主要体现在：

局限于数据中心内的应用场景：在跨地域或跨云应用中，Dubbo 的传统架构缺乏对广域环境的原生支持，导致开发者需要在多种协议和框架中切换，增加了复杂性。

南北向与东西向流量的双重挑战：在现代微服务架构下，传统的 RPC 框架往往更侧重南北向流量优化，而服务间（东西向）通信的性能要求日益增加，对传统 Dubbo 架构提出了新的挑战。

云原生与跨语言互操作性要求：随着云原生技术的普及，系统需要对 HTTP 协议有更深层次的支持，并具备跨语言的通信能力，然而传统 Dubbo 在这一点上并未原生优化。

Triple X 的变革和突破： Triple X 的诞生，直接回应了这些痛点，它不仅延续了 Dubbo 一贯的高性能通信能力，还实现了与 gRPC 协议的全面兼容，通过支持 HTTP/1、HTTP/2、HTTP/3 等协议，为跨云、跨区域的广泛应用场景提供了更具灵活性和高效性的解决方案。

在这里插入图片描述

Triple X 核心能力概述

全面支持南北向与东西向流量：Triple X 无缝支持从客户端到服务器的南北向流量及服务间通信的东西向流量，并确保灵活转换，提升通信链路的整体效率。

遵循 gRPC 协议标准：Triple X 遵循了 gRPC 协议标准，支持通过 Protobuf 进行通信，与 gRPC 服务无缝交互，扩展了 Dubbo 的跨语言、跨平台通信能力。

基于 HTTP 协议，原生支持云原生架构：Triple X 构建于 HTTP/1、HTTP/2 和 HTTP/3 协议栈之上，全面优化了网络通信性能，并与现代云原生基础设施无缝集成，支持各种网关和服务网格。

高性能优化：Triple X 提供了极致的性能提升，尤其在高并发和弱网环境下表现卓越，极大提高了系统吞吐量与响应速度。

平滑迁移与框架兼容：支持从现有的 Spring Web 项目无缝迁移，开发者无需更改现有代码即可切换至 Triple X，提升性能，并保留对 Spring MVC 等框架的支持。

高扩展性：Triple X 新引入 20+ SPI 扩展点，支持自定义核心行为，包括路由映射、参数解析、序列化及异常处理等。显著提升框架的灵活性和可定制性，使开发者能够根据特定业务需求和场景自定义行为，提高开发效率并降低定制化成本。

ChannelPipeline

org.apache.dubbo.remoting.transport.netty4.NettyConnectionClient
org.apache.dubbo.remoting.transport.netty4.NettyConnectionClient#initBootstrap
org.apache.dubbo.rpc.protocol.tri.TripleHttp2Protocol#configClientPipeline
io.netty.handler.codec.http2.Http2MultiplexHandler

Dubbo 使用的是Netty框架，Http2 Stream 多路复用，而Netty 对于每一个Stream 都会创建一个子Channel,子Channel 也会有自己的 ChannelPipeline

在一个Connection上，ChannelPipeline的配置如下：

SslClientTlsHandler(可选 加密传输)
Http2FrameCodec HTTP2 Frame编解码器
Http2MultiplexHandler HTTP2多路复用的支持
TripleGoAwayHandler http2 GoAway 帧（类型0x7） 用于启动链接关闭或发出严重错误状态信号
TripleTailHandler 避免内存泄漏

Http2FrameCodec codec = Http2FrameCodecBuilder.forClient()
        handlers.add(new ChannelHandlerPretender(codec));
        handlers.add(new ChannelHandlerPretender(new Http2MultiplexHandler(new ChannelDuplexHandler())));
        handlers.add(new ChannelHandlerPretender(new TriplePingPongHandler(UrlUtils.getCloseTimeout(url))));
        handlers.add(new ChannelHandlerPretender(new TripleGoAwayHandler()));
        handlers.add(new ChannelHandlerPretender(new TripleTailHandler()));

org.apache.dubbo.remoting.transport.netty4.NettyConnectionClient#initBootstrap
protected void initBootstrap() {
        Bootstrap bootstrap = new Bootstrap();
        bootstrap
                .group(NettyEventLoopFactory.NIO_EVENT_LOOP_GROUP.get())
                .option(ChannelOption.SO_KEEPALIVE, true)
                .option(ChannelOption.TCP_NODELAY, true)
                .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)

        bootstrap.handler(new ChannelInitializer<SocketChannel>() {
            @Override
            protected void initChannel(SocketChannel ch) {
                NettyChannel nettyChannel = NettyChannel.getOrAddChannel(ch, getUrl(), getChannelHandler());
                ChannelPipeline pipeline = ch.pipeline();

                if (sslContext != null) {
                    pipeline.addLast("negotiation", new SslClientTlsHandler(sslContext));
                }

                //TripleHttp2Protocol
                protocol.configClientPipeline(getUrl(), operator, nettySslContextOperator);
            }
        });
        this.bootstrap = bootstrap;
    }

org.apache.dubbo.rpc.protocol.tri.TripleHttp2Protocol#configClientPipeline

    @Override
    public void configClientPipeline(URL url, ChannelOperator operator, ContextOperator contextOperator) {
        TripleConfig tripleConfig = ConfigManager.getProtocolOrDefault(url).getTripleOrDefault();
        Http2FrameCodec codec = Http2FrameCodecBuilder.forClient()
                .gracefulShutdownTimeoutMillis(10000)
                .initialSettings(new Http2Settings()
                        .headerTableSize(tripleConfig.getHeaderTableSizeOrDefault())
                        .pushEnabled(tripleConfig.getEnablePushOrDefault())
                        .maxConcurrentStreams(tripleConfig.getMaxConcurrentStreamsOrDefault())
                        .initialWindowSize(tripleConfig.getInitialWindowSizeOrDefault())
                        .maxFrameSize(tripleConfig.getMaxFrameSizeOrDefault())
                        .maxHeaderListSize(tripleConfig.getMaxHeaderListSizeOrDefault()))
                .frameLogger(CLIENT_LOGGER)
                .validateHeaders(false)
                .build();
        //        codec.connection().local().flowController().frameWriter(codec.encoder().frameWriter());
        List<ChannelHandler> handlers = new ArrayList<>();
        handlers.add(new ChannelHandlerPretender(codec));
        handlers.add(new ChannelHandlerPretender(new Http2MultiplexHandler(new ChannelDuplexHandler())));
        handlers.add(new ChannelHandlerPretender(new TriplePingPongHandler(UrlUtils.getCloseTimeout(url))));
        handlers.add(new ChannelHandlerPretender(new TripleGoAwayHandler()));
        handlers.add(new ChannelHandlerPretender(new TripleTailHandler()));
        operator.configChannelHandler(handlers);
    }

在Stream 上，ChannelPipeline的配置如下：

TripleCommandOutBoundHandler 命令输出
TripleHttp2ClientResponseHandler 对服务端响应的处理

org.apache.dubbo.rpc.protocol.tri.h12.http2.Http2TripleClientStream#initStreamChannel

    @Override
    protected TripleStreamChannelFuture initStreamChannel(Channel parent) {
        Http2StreamChannelBootstrap bootstrap = new Http2StreamChannelBootstrap(parent);
        bootstrap.handler(new ChannelInboundHandlerAdapter() {
            @Override
            public void handlerAdded(ChannelHandlerContext ctx) {
                ctx.channel()
                        .pipeline()
                        .addLast(new TripleCommandOutBoundHandler())
                        .addLast(new TripleHttp2ClientResponseHandler(createTransportListener()));
            }
        });
        TripleStreamChannelFuture streamChannelFuture = new TripleStreamChannelFuture(parent);
        writeQueue.enqueue(CreateStreamQueueCommand.create(bootstrap, streamChannelFuture));
        return streamChannelFuture;
    }```



## Triple协议TripleProtocol

消费方的调用流程体系，首先得引用服务，创建对应的Invoker.
Triple服务的引用代码 
org.apache.dubbo.rpc.protocol.tri.TripleProtocol#refer

```java
    @Override
    public <T> Invoker<T> refer(Class<T> type, URL url) throws RpcException {
        optimizeSerialization(url);
        ExecutorService streamExecutor = getOrCreateStreamExecutor(url.getOrDefaultApplicationModel(), url);
        AbstractConnectionClient connectionClient = Http3Exchanger.isEnabled(url)
                ? Http3Exchanger.connect(url)
                : PortUnificationExchanger.connect(url, new DefaultPuHandler());
        TripleInvoker<T> invoker =
                new TripleInvoker<>(type, url, acceptEncodings, connectionClient, invokers, streamExecutor);
        invokers.add(invoker);
        return invoker;
    }

服务引用

服务引用
org.apache.dubbo.rpc.protocol.tri.TripleProtocol#refer 服务引用
 org.apache.dubbo.rpc.protocol.tri.TripleInvoker#TripleInvoker 实例化Invoker
   org.apache.dubbo.remoting.transport.netty4.NettyConnectionManager#connect 实例化Connection
     org.apache.dubbo.remoting.transport.netty4.NettyConnectionClient#initBootstrap 构建Bootstrap
       org.apache.dubbo.remoting.transport.AbstractClient#connect 建立连接

服务调用

org.apache.dubbo.rpc.protocol.tri.TripleInvoker#doInvoke Triple协议调用
org.apache.dubbo.rpc.protocol.tri.TripleInvoker#invokeUnary Unary调用
org.apache.dubbo.rpc.protocol.tri.TripleInvoker#createRequest 请求元数据
org.apache.dubbo.rpc.protocol.tri.stream.ClientStreamFactory#createClientStream 开启stream
org.apache.dubbo.rpc.protocol.tri.h12.http2.Http2TripleClientStream 开启stream
org.apache.dubbo.rpc.protocol.tri.observer.ClientCallToObserverAdapter#onNext 观测适配器
org.apache.dubbo.rpc.protocol.tri.call.TripleClientCall#sendMessage 发送消息
org.apache.dubbo.rpc.protocol.tri.stream.Stream#sendHeader 发送Header 帧
org.apache.dubbo.rpc.model.PackableMethod#packRequest 打包消息序列化
org.apache.dubbo.rpc.protocol.tri.stream.ClientStream#sendMessage 发送DATA帧
org.apache.dubbo.rpc.protocol.tri.observer.ClientCallToObserverAdapter#onCompleted 完成事件
org.apache.dubbo.rpc.protocol.tri.call.ClientCall#halfClose 发送endSteam Stream半关闭状态

服务调用完整堆栈

halfClose:229, TripleClientCall (org.apache.dubbo.rpc.protocol.tri.call)
onCompleted:55, ClientCallToObserverAdapter (org.apache.dubbo.rpc.protocol.tri.observer)
invokeUnary:284, TripleInvoker (org.apache.dubbo.rpc.protocol.tri)
doInvoke:171, TripleInvoker (org.apache.dubbo.rpc.protocol.tri)
doInvokeAndReturn:249, AbstractInvoker (org.apache.dubbo.rpc.protocol)
invoke:192, AbstractInvoker (org.apache.dubbo.rpc.protocol)
invoke:71, ListenerInvokerWrapper (org.apache.dubbo.rpc.listener)
invoke:40, RpcExceptionFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:96, ConsumingFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:106, ReferenceCountInvokerWrapper (org.apache.dubbo.rpc.protocol)
invoke:800, ServiceDiscoveryRegistryDirectory$InstanceWrappedInvoker (org.apache.dubbo.registry.client)
invokeWithContext:412, AbstractClusterInvoker (org.apache.dubbo.rpc.cluster.support)
doInvoke:82, FailoverClusterInvoker (org.apache.dubbo.rpc.cluster.support)
invoke:366, AbstractClusterInvoker (org.apache.dubbo.rpc.cluster.support)
invoke:46, RouterSnapshotFilter (org.apache.dubbo.rpc.cluster.router)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:109, MonitorFilter (org.apache.dubbo.monitor.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:57, MetricsClusterFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:53, FutureFilter (org.apache.dubbo.rpc.protocol.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:40, ContextHolderParametersSelectedTransferFilter (org.apache.dubbo.spring.security.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:86, MetricsFilter (org.apache.dubbo.metrics.filter)
invoke:38, MetricsConsumerFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:40, ConsumerClassLoaderFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:60, ObservationSenderFilter (org.apache.dubbo.tracing.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:119, ConsumerContextFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:101, AbstractCluster$ClusterFilterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:107, MockClusterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:171, ScopeClusterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:296, MigrationInvoker (org.apache.dubbo.registry.client.migration)
invoke:64, InvocationUtil (org.apache.dubbo.rpc.proxy)
invoke:81, InvokerInvocationHandler (org.apache.dubbo.rpc.proxy)
getAllCity:-1, CityServiceDubboProxy2 (cn.itbox.uap.sys.setting.interfaces.service)
invokeInterface:-1, LambdaForm$DMH/0x000000c001018400 (java.lang.invoke)
invoke:-1, LambdaForm$MH/0x000000c0010ac000 (java.lang.invoke)
invokeExact_MT:-1, Invokers$Holder (java.lang.invoke)
invokeImpl:153, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:103, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:580, Method (java.lang.reflect)
invoke:54, LazyTargetInvocationHandler (org.apache.dubbo.config.spring.util)
getAllCity:-1, CityServiceDubboProxy2 (cn.itbox.uap.sys.setting.interfaces.service)
invokeVirtual:-1, DirectMethodHandle$Holder (java.lang.invoke)
invoke:-1, LambdaForm$MH/0x000000c0010ac000 (java.lang.invoke)
invokeExact_MT:-1, Invokers$Holder (java.lang.invoke)
invokeImpl:153, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:103, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:580, Method (java.lang.reflect)
invokeJoinpointUsingReflection:343, AopUtils (org.springframework.aop.support)
invokeJoinpoint:196, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:163, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:750, CglibAopProxy$CglibMethodInvocation (org.springframework.aop.framework)
invoke:141, MethodValidationInterceptor (org.springframework.validation.beanvalidation)
proceed:184, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:750, CglibAopProxy$CglibMethodInvocation (org.springframework.aop.framework)
intercept:702, CglibAopProxy$DynamicAdvisedInterceptor (org.springframework.aop.framework)
getAllCity:-1, CityServiceDubboProxy2$$SpringCGLIB$$0 (cn.itbox.uap.sys.setting.interfaces.service)
getAllCity:70, ControllerToDubboServiceImpl (cn.itbox.uap.demo.org.application.service.impl)
invokeMethod:-1, ControllerToDubboServiceImplDubboWrap3 (cn.itbox.uap.demo.org.application.service.impl)
doInvoke:89, JavassistProxyFactory$1 (org.apache.dubbo.rpc.proxy.javassist)
invoke:100, AbstractProxyInvoker (org.apache.dubbo.rpc.proxy)
invoke:55, DelegateProviderMetaDataInvoker (org.apache.dubbo.config.invoker)
invoke:56, InvokerWrapper (org.apache.dubbo.rpc.protocol)
invoke:38, ClassLoaderCallbackFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:37, HttpContextCallbackFilter (org.apache.dubbo.rpc.protocol.tri.h12)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
lambda$invoke$0:78, RestExtensionExecutionFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
get:-1, RestExtensionExecutionFilter$$Lambda/0x000000c001c3d680 (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:62, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:28, RestFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:58, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
execute:51, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:81, RestExtensionExecutionFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:36, RestFilterAdapter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:131, ProvidingFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:80, TraceFilter (org.apache.dubbo.rpc.protocol.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:45, TimeoutFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:109, MonitorFilter (org.apache.dubbo.monitor.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:55, ExceptionFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:120, AccessLogFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:67, GlobalDubboHandlerFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:222, GenericFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:50, HttpContextFilter (org.apache.dubbo.rpc.protocol.tri.h12)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:54, ClassLoaderFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:41, EchoFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:86, MetricsFilter (org.apache.dubbo.metrics.filter)
invoke:37, MetricsProviderFilter (org.apache.dubbo.metrics.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:59, ObservationReceiverFilter (org.apache.dubbo.tracing.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:66, ProfilerServerFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:191, ContextFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:69, AbstractServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12)
onComplete:50, UnaryServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12)
onMessage:120, DefaultHttp11ServerTransportListener$AutoCompleteUnaryServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12.http1)
onMessage:-1, DefaultHttp11ServerTransportListener$$Lambda/0x000000c001c09860 (org.apache.dubbo.rpc.protocol.tri.h12.http1)
decode:42, DefaultListeningDecoder (org.apache.dubbo.remoting.http12.message)
onMessage:39, DefaultHttpMessageListener (org.apache.dubbo.rpc.protocol.tri.h12)
doOnData:186, AbstractServerTransportListener (org.apache.dubbo.rpc.protocol.tri.h12)
lambda$onData$1:171, AbstractServerTransportListener (org.apache.dubbo.rpc.protocol.tri.h12)
run:-1, AbstractServerTransportListener$$Lambda/0x000000c001c02cf8 (org.apache.dubbo.rpc.protocol.tri.h12)
run:105, SerializingExecutor (org.apache.dubbo.common.threadpool.serial)
runWorker:1144, ThreadPoolExecutor (java.util.concurrent)
run:642, ThreadPoolExecutor$Worker (java.util.concurrent)
run:39, InternalRunnable (org.apache.dubbo.common.threadlocal)
runWith:1596, Thread (java.lang)
run:1583, Thread (java.lang)

处理响应

处理响应
org.apache.dubbo.rpc.protocol.tri.transport.TripleHttp2ClientResponseHandler#channelRead0
  org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#onHeader
org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#onHeaderReceived 接收Headers
  org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#onData 接收Data帧
org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#doOnData  
    org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder#deframe 解帧
      org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder.Listener#onRawMessage
      org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder.Listener#onRawMessage
      org.apache.dubbo.rpc.protocol.tri.stream.Stream.Listener#onMessage
       org.apache.dubbo.rpc.model.PackableMethod#parseResponse 消息解包 反序列化
        org.apache.dubbo.rpc.protocol.tri.DeadlineFuture#received 受到响应结果 线程唤醒

完整堆栈

received:80, DeadlineFuture (org.apache.dubbo.rpc.protocol.tri)
onClose:56, UnaryClientCallListener (org.apache.dubbo.rpc.protocol.tri.call)
onComplete:126, TripleClientCall (org.apache.dubbo.rpc.protocol.tri.call)
finishProcess:224, AbstractTripleClientStream$ClientTransportListener (org.apache.dubbo.rpc.protocol.tri.stream)
close:291, AbstractTripleClientStream$ClientTransportListener$1 (org.apache.dubbo.rpc.protocol.tri.stream)
deliver:101, TriDecoder (org.apache.dubbo.rpc.protocol.tri.frame)
close:67, TriDecoder (org.apache.dubbo.rpc.protocol.tri.frame)
onTrailersReceived:313, AbstractTripleClientStream$ClientTransportListener (org.apache.dubbo.rpc.protocol.tri.stream)
lambda$onHeader$1:412, AbstractTripleClientStream$ClientTransportListener (org.apache.dubbo.rpc.protocol.tri.stream)
run:-1, AbstractTripleClientStream$ClientTransportListener$$Lambda/0x000000c001c6e898 (org.apache.dubbo.rpc.protocol.tri.stream)
run:105, SerializingExecutor (org.apache.dubbo.common.threadpool.serial)
run:151, ThreadlessExecutor$RunnableWrapper (org.apache.dubbo.common.threadpool)
waitAndDrain:77, ThreadlessExecutor (org.apache.dubbo.common.threadpool)
get:220, AsyncRpcResult (org.apache.dubbo.rpc)
waitForResultIfSync:293, AbstractInvoker (org.apache.dubbo.rpc.protocol)
invoke:195, AbstractInvoker (org.apache.dubbo.rpc.protocol)
invoke:71, ListenerInvokerWrapper (org.apache.dubbo.rpc.listener)
invoke:40, RpcExceptionFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:96, ConsumingFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:106, ReferenceCountInvokerWrapper (org.apache.dubbo.rpc.protocol)
invoke:800, ServiceDiscoveryRegistryDirectory$InstanceWrappedInvoker (org.apache.dubbo.registry.client)
invokeWithContext:412, AbstractClusterInvoker (org.apache.dubbo.rpc.cluster.support)
doInvoke:82, FailoverClusterInvoker (org.apache.dubbo.rpc.cluster.support)
invoke:366, AbstractClusterInvoker (org.apache.dubbo.rpc.cluster.support)
invoke:46, RouterSnapshotFilter (org.apache.dubbo.rpc.cluster.router)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:109, MonitorFilter (org.apache.dubbo.monitor.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:57, MetricsClusterFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:53, FutureFilter (org.apache.dubbo.rpc.protocol.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:40, ContextHolderParametersSelectedTransferFilter (org.apache.dubbo.spring.security.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:86, MetricsFilter (org.apache.dubbo.metrics.filter)
invoke:38, MetricsConsumerFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:40, ConsumerClassLoaderFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:60, ObservationSenderFilter (org.apache.dubbo.tracing.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:119, ConsumerContextFilter (org.apache.dubbo.rpc.cluster.filter.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:101, AbstractCluster$ClusterFilterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:107, MockClusterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:171, ScopeClusterInvoker (org.apache.dubbo.rpc.cluster.support.wrapper)
invoke:296, MigrationInvoker (org.apache.dubbo.registry.client.migration)
invoke:64, InvocationUtil (org.apache.dubbo.rpc.proxy)
invoke:81, InvokerInvocationHandler (org.apache.dubbo.rpc.proxy)
getAllCity:-1, CityServiceDubboProxy2 (cn.itbox.uap.sys.setting.interfaces.service)
invokeInterface:-1, LambdaForm$DMH/0x000000c001018400 (java.lang.invoke)
invoke:-1, LambdaForm$MH/0x000000c0010ac000 (java.lang.invoke)
invokeExact_MT:-1, Invokers$Holder (java.lang.invoke)
invokeImpl:153, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:103, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:580, Method (java.lang.reflect)
invoke:54, LazyTargetInvocationHandler (org.apache.dubbo.config.spring.util)
getAllCity:-1, CityServiceDubboProxy2 (cn.itbox.uap.sys.setting.interfaces.service)
invokeVirtual:-1, DirectMethodHandle$Holder (java.lang.invoke)
invoke:-1, LambdaForm$MH/0x000000c0010ac000 (java.lang.invoke)
invokeExact_MT:-1, Invokers$Holder (java.lang.invoke)
invokeImpl:153, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:103, DirectMethodHandleAccessor (jdk.internal.reflect)
invoke:580, Method (java.lang.reflect)
invokeJoinpointUsingReflection:343, AopUtils (org.springframework.aop.support)
invokeJoinpoint:196, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:163, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:750, CglibAopProxy$CglibMethodInvocation (org.springframework.aop.framework)
invoke:141, MethodValidationInterceptor (org.springframework.validation.beanvalidation)
proceed:184, ReflectiveMethodInvocation (org.springframework.aop.framework)
proceed:750, CglibAopProxy$CglibMethodInvocation (org.springframework.aop.framework)
intercept:702, CglibAopProxy$DynamicAdvisedInterceptor (org.springframework.aop.framework)
getAllCity:-1, CityServiceDubboProxy2$$SpringCGLIB$$0 (cn.itbox.uap.sys.setting.interfaces.service)
getAllCity:70, ControllerToDubboServiceImpl (cn.itbox.uap.demo.org.application.service.impl)
invokeMethod:-1, ControllerToDubboServiceImplDubboWrap3 (cn.itbox.uap.demo.org.application.service.impl)
doInvoke:89, JavassistProxyFactory$1 (org.apache.dubbo.rpc.proxy.javassist)
invoke:100, AbstractProxyInvoker (org.apache.dubbo.rpc.proxy)
invoke:55, DelegateProviderMetaDataInvoker (org.apache.dubbo.config.invoker)
invoke:56, InvokerWrapper (org.apache.dubbo.rpc.protocol)
invoke:38, ClassLoaderCallbackFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:37, HttpContextCallbackFilter (org.apache.dubbo.rpc.protocol.tri.h12)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
lambda$invoke$0:78, RestExtensionExecutionFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
get:-1, RestExtensionExecutionFilter$$Lambda/0x000000c001c3d680 (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:62, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:28, RestFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
doFilter:58, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
execute:51, DefaultFilterChain (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:81, RestExtensionExecutionFilter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:36, RestFilterAdapter (org.apache.dubbo.rpc.protocol.tri.rest.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:131, ProvidingFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:80, TraceFilter (org.apache.dubbo.rpc.protocol.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:45, TimeoutFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:109, MonitorFilter (org.apache.dubbo.monitor.support)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:55, ExceptionFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:120, AccessLogFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:67, GlobalDubboHandlerFilter (cn.itbox.uap.dubbo.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:222, GenericFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:50, HttpContextFilter (org.apache.dubbo.rpc.protocol.tri.h12)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:54, ClassLoaderFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:41, EchoFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:86, MetricsFilter (org.apache.dubbo.metrics.filter)
invoke:37, MetricsProviderFilter (org.apache.dubbo.metrics.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:59, ObservationReceiverFilter (org.apache.dubbo.tracing.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:66, ProfilerServerFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:191, ContextFilter (org.apache.dubbo.rpc.filter)
invoke:349, FilterChainBuilder$CopyOfFilterChainNode (org.apache.dubbo.rpc.cluster.filter)
invoke:197, FilterChainBuilder$CallbackRegistrationInvoker (org.apache.dubbo.rpc.cluster.filter)
invoke:69, AbstractServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12)
onComplete:50, UnaryServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12)
onMessage:120, DefaultHttp11ServerTransportListener$AutoCompleteUnaryServerCallListener (org.apache.dubbo.rpc.protocol.tri.h12.http1)
onMessage:-1, DefaultHttp11ServerTransportListener$$Lambda/0x000000c001c09860 (org.apache.dubbo.rpc.protocol.tri.h12.http1)
decode:42, DefaultListeningDecoder (org.apache.dubbo.remoting.http12.message)
onMessage:39, DefaultHttpMessageListener (org.apache.dubbo.rpc.protocol.tri.h12)
doOnData:186, AbstractServerTransportListener (org.apache.dubbo.rpc.protocol.tri.h12)
lambda$onData$1:171, AbstractServerTransportListener (org.apache.dubbo.rpc.protocol.tri.h12)
run:-1, AbstractServerTransportListener$$Lambda/0x000000c001c02cf8 (org.apache.dubbo.rpc.protocol.tri.h12)
run:105, SerializingExecutor (org.apache.dubbo.common.threadpool.serial)
runWorker:1144, ThreadPoolExecutor (java.util.concurrent)
run:642, ThreadPoolExecutor$Worker (java.util.concurrent)
run:39, InternalRunnable (org.apache.dubbo.common.threadlocal)
runWith:1596, Thread (java.lang)
run:1583, Thread (java.lang)

TripleInvoker

Triple 协议服务的引用过程，就是创建 TripleInvoker 的过程。
在 TripleInvoker 的构造函数中，会创建 Connection 连接对象。在 Connection 的构造函数中，又会创建 Bootstrap。
org.apache.dubbo.remoting.transport.netty4.NettyConnectionClient#initBootstrap()

@Override
    protected void initConnectionClient() {
        protocol = getUrl().getOrDefaultFrameworkModel()
                .getExtensionLoader(WireProtocol.class)
                .getExtension(getUrl().getProtocol());
        super.initConnectionClient();
    }

    protected void initBootstrap() {
        Bootstrap bootstrap = new Bootstrap();
        bootstrap
                .group(NettyEventLoopFactory.NIO_EVENT_LOOP_GROUP.get())
                .option(ChannelOption.SO_KEEPALIVE, true)
                .option(ChannelOption.TCP_NODELAY, true)
                .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
                .remoteAddress(getConnectAddress())
                .channel(socketChannelClass());

        NettyConnectionHandler connectionHandler = new NettyConnectionHandler(this);
        bootstrap.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, getConnectTimeout());
        SslContext sslContext = SslContexts.buildClientSslContext(getUrl());
        bootstrap.handler(new ChannelInitializer<SocketChannel>() {
            @Override
            protected void initChannel(SocketChannel ch) {
                NettyChannel nettyChannel = NettyChannel.getOrAddChannel(ch, getUrl(), getChannelHandler());
                ChannelPipeline pipeline = ch.pipeline();
                NettySslContextOperator nettySslContextOperator = new NettySslContextOperator();

                if (sslContext != null) {
                    pipeline.addLast("negotiation", new SslClientTlsHandler(sslContext));
                }

                //                pipeline.addLast("logging", new LoggingHandler(LogLevel.INFO)); //for debug

                int heartbeat = UrlUtils.getHeartbeat(getUrl());
                pipeline.addLast("client-idle-handler", new IdleStateHandler(heartbeat, 0, 0, MILLISECONDS));

                pipeline.addLast(Constants.CONNECTION_HANDLER_NAME, connectionHandler);

                NettyConfigOperator operator = new NettyConfigOperator(nettyChannel, getChannelHandler());
                protocol.configClientPipeline(getUrl(), operator, nettySslContextOperator);
                // set null but do not close this client, it will be reconnecting in the future
                ch.closeFuture().addListener(channelFuture -> clearNettyChannel());
                // TODO support Socks5
            }
        });
        this.bootstrap = bootstrap;
    }

调用TripleHttp2Protocol#configClientPipeline()对 ChannelPipeline 进行配置。
org.apache.dubbo.rpc.protocol.tri.TripleHttp2Protocol#configClientPipeline


    @Override
    public void configClientPipeline(URL url, ChannelOperator operator, ContextOperator contextOperator) {
        TripleConfig tripleConfig = ConfigManager.getProtocolOrDefault(url).getTripleOrDefault();
        Http2FrameCodec codec = Http2FrameCodecBuilder.forClient()
                .gracefulShutdownTimeoutMillis(10000)
                .initialSettings(new Http2Settings()
                        .headerTableSize(tripleConfig.getHeaderTableSizeOrDefault())
                        .pushEnabled(tripleConfig.getEnablePushOrDefault())
                        .maxConcurrentStreams(tripleConfig.getMaxConcurrentStreamsOrDefault())
                        .initialWindowSize(tripleConfig.getInitialWindowSizeOrDefault())
                        .maxFrameSize(tripleConfig.getMaxFrameSizeOrDefault())
                        .maxHeaderListSize(tripleConfig.getMaxHeaderListSizeOrDefault()))
                .frameLogger(CLIENT_LOGGER)
                .validateHeaders(false)
                .build();
        //Triple在初次建立连接时，通过 TripleHttpProtocol 初始化 HTTP/2 配置，默认流控窗口 DEFAULT_WINDOW_INIT_SIZE = MIB_8，并在服务端和客户端加入自己的 outbound 流控接口。
        //参考：https://www.jianshu.com/p/95868974700a    
        codec.connection().local().flowController().frameWriter(codec.encoder().frameWriter());
        List<ChannelHandler> handlers = new ArrayList<>();
        handlers.add(new ChannelHandlerPretender(codec));
        handlers.add(new ChannelHandlerPretender(new Http2MultiplexHandler(new ChannelDuplexHandler())));
        handlers.add(new ChannelHandlerPretender(new TriplePingPongHandler(UrlUtils.getCloseTimeout(url))));
        handlers.add(new ChannelHandlerPretender(new TripleGoAwayHandler()));
        handlers.add(new ChannelHandlerPretender(new TripleTailHandler()));
        operator.configChannelHandler(handlers);
    }

Http2FrameCodec 是 Netty 提供的 HTTP2 Frame的编解码器，让开发者不用关心 Frame 的细节。
Http2MultiplexHandler 是 Netty 对 HTTP2 多路复用的支持，给每个 Stream 也创建了一个 Channel。

建立连接后，发起 Triple RPC 调用，实际触发的是TripleInvoker#doInvoke()。

从 Invocation 解析出 ServiceDescriptor 和 MethodDescriptor，明确要调用的服务和方法。
实例化 TripleClientCall，用于发起客户端调用。
根据 RPC 调用类型执行不同方法，最常用的是 UNARY，就像调用本地方法一样，线程会阻塞等待结果。

@Override
    protected Result doInvoke(final Invocation invocation) {
    //链接校验
        if (!connectionClient.isConnected()) {
            CompletableFuture<AppResponse> future = new CompletableFuture<>();
            RpcException exception = TriRpcStatus.UNAVAILABLE
                    .withDescription(String.format("upstream %s is unavailable", getUrl().getAddress()))
                    .asException();
            future.completeExceptionally(exception);
            return new AsyncRpcResult(future, invocation);
        }

        ConsumerModel consumerModel = (ConsumerModel)
                (invocation.getServiceModel() != null ? invocation.getServiceModel() : getUrl().getServiceModel());
                //服务描述符
        ServiceDescriptor serviceDescriptor = consumerModel.getServiceModel();
        //方法描述符
        MethodDescriptor methodDescriptor =
                serviceDescriptor.getMethod(invocation.getMethodName(), invocation.getParameterTypes());
                //泛化调用
        if (methodDescriptor == null
                && RpcUtils.isGenericCall(
                        ((RpcInvocation) invocation).getParameterTypesDesc(), invocation.getMethodName())) {
            // Only reach when server generic
            methodDescriptor = ServiceDescriptorInternalCache.genericService()
                    .getMethod(invocation.getMethodName(), invocation.getParameterTypes());
        }
        ExecutorService callbackExecutor =
                isSync(methodDescriptor, invocation) ? new ThreadlessExecutor() : streamExecutor;
                //客户端调用对象
        ClientCall call = new TripleClientCall(
                connectionClient, callbackExecutor, getUrl().getOrDefaultFrameworkModel(), writeQueue);
        RpcContext.getServiceContext().setLocalAddress(connectionClient.getLocalAddress());
        AsyncRpcResult result;
        try {
            switch (methodDescriptor.getRpcType()) {
            // RPC调用类型 支持4种 UNARY最常用
                case UNARY:
                    result = invokeUnary(methodDescriptor, invocation, call, callbackExecutor);
                    break;
                case SERVER_STREAM:
                    result = invokeServerStream(methodDescriptor, invocation, call);
                    break;
                case CLIENT_STREAM:
                case BI_STREAM:
                    result = invokeBiOrClientStream(methodDescriptor, invocation, call);
                    break;
                default:
                    throw new IllegalStateException("Can not reach here");
            }
            return result;
        } catch (Throwable t) {
            final TriRpcStatus status =
                    TriRpcStatus.INTERNAL.withCause(t).withDescription("Call aborted cause client exception");
            RpcException e = status.asException();
            try {
                call.cancelByLocal(e);
            } catch (Throwable t1) {
                LOGGER.error(PROTOCOL_FAILED_REQUEST, "", "", "Cancel triple request failed", t1);
            }
            CompletableFuture<AppResponse> future = new CompletableFuture<>();
            future.completeExceptionally(e);
            return new AsyncRpcResult(future, invocation);
        }
    }

以最常用的 UNARY 调用为例：

基于 Invocation 创建请求元数据对象 RequestMetadata。
因为网络调用是异步的，创建一个 DeadlineFuture 等待结果。
开启Stream、发送请求、发送endStream，请求结束。

    AsyncRpcResult invokeUnary(
            MethodDescriptor methodDescriptor,
            Invocation invocation,
            ClientCall call,
            ExecutorService callbackExecutor) {

//超时时间
        int timeout = RpcUtils.calculateTimeout(getUrl(), invocation, RpcUtils.getMethodName(invocation), 3000);
        if (timeout <= 0) {
            return AsyncRpcResult.newDefaultAsyncResult(
                    new RpcException(
                            RpcException.TIMEOUT_TERMINATE,
                            "No time left for making the following call: " + invocation.getServiceName() + "."
                                    + RpcUtils.getMethodName(invocation) + ", terminate directly."),
                    invocation);
        }
        invocation.setAttachment(TIMEOUT_KEY, String.valueOf(timeout));

//回调线程池
        final AsyncRpcResult result;
        //网络调用是异步的，创建一个DeadlineFuture等待结果
        DeadlineFuture future = DeadlineFuture.newFuture(
                getUrl().getPath(), methodDescriptor.getMethodName(), getUrl().getAddress(), timeout, callbackExecutor);

//创建请求元数据
        RequestMetadata request = createRequest(methodDescriptor, invocation, timeout);

//实参
        final Object pureArgument;

        if (methodDescriptor instanceof StubMethodDescriptor) {
            pureArgument = invocation.getArguments()[0];
        } else {
            pureArgument = invocation.getArguments();
        }
        result = new AsyncRpcResult(future, invocation);
        if (setFutureWhenSync || ((RpcInvocation) invocation).getInvokeMode() != InvokeMode.SYNC) {
            FutureContext.getContext().setCompatibleFuture(future);
        }

        result.setExecutor(callbackExecutor);
        ClientCall.Listener callListener = new UnaryClientCallListener(future);

//开启stream发起调用
        final StreamObserver<Object> requestObserver = call.start(request, callListener);
        requestObserver.onNext(pureArgument);
        requestObserver.onCompleted();
        return result;
    }

发送请求的方法是org.apache.dubbo.rpc.protocol.tri.call.TripleClientCall#sendMessage

发送Header 帧
请求消息打包
发送Data 帧

@Override
    public void sendMessage(Object message) {
        if (canceled && null != streamException) {
            throw new IllegalStateException("Due flowcontrol over pendingbytes, Call already canceled");
        } else if (canceled) {
            throw new IllegalStateException("Call already canceled");
        }
        // 先发送Headers帧，再发送Data帧
        if (!headerSent) {
            headerSent = true;
            stream.sendHeader(requestMetadata.toHeaders());
        }
        final byte[] data;
        try {
        // 消息打包 Protobuf 或者 Wrap
            data = requestMetadata.packableMethod.packRequest(message);
             // 是否要压缩
            int compressed = Identity.MESSAGE_ENCODING.equals(requestMetadata.compressor.getMessageEncoding()) ? 0 : 1;
            final byte[] compress = requestMetadata.compressor.compress(data);
             // 发送Data帧
            stream.sendMessage(compress, compressed).addListener(f -> {
                if (!f.isSuccess()) {
                    cancelByLocal(f.cause());
                }
            });
        } catch (Throwable t) {
          //此处省略
        }
    }

请求到这一步还没完，没发送 endStream 服务端不知道请求已经结束。所以紧接着还要调用ClientCallToObserverAdapter#onCompleted()来发送一个 endStream 消息代表请求结束。

org.apache.dubbo.rpc.protocol.tri.observer.ClientCallToObserverAdapter#onCompleted

    @Override
    public void onCompleted() {
        if (terminated) {
            return;
        }
        // 流半关闭状态 即发送endStream
        call.halfClose();
        this.terminated = true;
    }

所谓的流半关闭，其实就是发送一个空的Data帧，只是 endStream 标记为true。

org.apache.dubbo.rpc.protocol.tri.call.TripleClientCall#halfClose
org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream#halfClose

    @Override
    public void halfClose() {
        if (!headerSent) {
            return;
        }
        if (canceled) {
            return;
        }
        stream.halfClose().addListener(f -> {
            if (!f.isSuccess()) {
                cancelByLocal(new IllegalStateException("Half close failed", f.cause()));
            }
        });
    }

    @Override
    public ChannelFuture halfClose() {
        ChannelFuture checkResult = preCheck();
        if (!checkResult.isSuccess()) {
            return checkResult;
        }
    // 发送一个空的Data帧 endStream=true
        final EndStreamQueueCommand cmd = EndStreamQueueCommand.create(streamChannelFuture);
        return this.writeQueue.enqueueFuture(cmd, parent.eventLoop()).addListener(future -> {
            if (future.isSuccess()) {
                halfClosed = true;
            }
        });
    }

至此，请求就结束了，客户端只要坐等服务端响应即可。

对服务端响应结果的处理器是TripleHttp2ClientResponseHandler：
org.apache.dubbo.rpc.protocol.tri.transport.TripleHttp2ClientResponseHandler#channelRead0

    @Override
    protected void channelRead0(ChannelHandlerContext ctx, Http2StreamFrame msg) throws Exception {
        if (msg instanceof Http2HeadersFrame) {
            final Http2HeadersFrame headers = (Http2HeadersFrame) msg;
            transportListener.onHeader(headers.headers(), headers.isEndStream());
        } else if (msg instanceof Http2DataFrame) {
            final Http2DataFrame data = (Http2DataFrame) msg;
            transportListener.onData(data.content(), data.isEndStream());
        } else {
            super.channelRead(ctx, msg);
        }
    }

org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#onHeader

        @Override
        public void onHeader(Http2Headers headers, boolean endStream) {
            executor.execute(() -> {
                if (endStream) {
                    if (!halfClosed) {
                        Channel channel = streamChannelFuture.getNow();
                        if (channel.isActive() && !rst) {
                            writeQueue.enqueue(
                                    CancelQueueCommand.createCommand(streamChannelFuture, Http2Error.CANCEL));
                            rst = true;
                        }
                    }
                    onTrailersReceived(headers);
                } else {
                    onHeaderReceived(headers);
                }
            });
        }

org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#onData

        @Override
        public void onData(ByteBuf data, boolean endStream) {
            try {
                executor.execute(() -> doOnData(data, endStream));
            } catch (Throwable t) {
            }
        }

//org.apache.dubbo.rpc.protocol.tri.stream.AbstractTripleClientStream.ClientTransportListener#doOnData
        private void doOnData(ByteBuf data, boolean endStream) {
            deframer.deframe(data);
        }


//org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder#deframe
    @Override
    public void deframe(ByteBuf data) {
        if (closing || closed) {
            // ignored
            return;
        }
        accumulate.addComponent(true, data);
        deliver();
    }

org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder#deliver
 private void deliver() {
        // We can have reentrancy here when using a direct executor, triggered by calls to
        // request more messages. This is safe as we simply loop until pendingDelivers = 0
        if (inDelivery) {
            return;
        }
        inDelivery = true;
        try {
            // Process the uncompressed bytes.
            while (pendingDeliveries > 0 && hasEnoughBytes()) {
                switch (state) {
                    case HEADER:
                        processHeader();
                        break;
                    case PAYLOAD:
                        // Read the body and deliver the message.
                        processBody();

                        // Since we've delivered a message, decrement the number of pending
                        // deliveries remaining.
                        pendingDeliveries--;
                        break;
                    default:
                        throw new AssertionError("Invalid state: " + state);
                }
            }
            if (closing) {
                if (!closed) {
                    closed = true;
                    accumulate.clear();
                    accumulate.release();
                    listener.close();
                }
            }
        } finally {
            inDelivery = false;
        }
    }

RPC 方法返回的结果封装在 Data 帧里，交给 TriDecoder 按照 Protobuf 的方式解帧。
解帧后再交给 PackableMethod#parseResponse()解包消息，也就是反序列化，拿到结果对象。
最后把结果写入 DeadlineFuture，业务线程拿到结果后唤醒，继续执行，整个 RPC 调用结束。

org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder#processHeader

    /**
     * Processes the GRPC compression header which is composed of the compression flag and the outer
     * frame length.
     */
    private void processHeader() {
        int type = accumulate.readUnsignedByte();
        if ((type & RESERVED_MASK) != 0) {
            throw new RpcException("gRPC frame header malformed: reserved bits not zero");
        }
        compressedFlag = (type & COMPRESSED_FLAG_MASK) != 0;

        requiredLength = accumulate.readInt();

        // Continue reading the frame body.
        state = GrpcDecodeState.PAYLOAD;
    }

org.apache.dubbo.rpc.protocol.tri.frame.TriDecoder#processBody

  /**
     * Processes the GRPC message body, which depending on frame header flags may be compressed.
     */
    private void processBody() {
        // There is no reliable way to get the uncompressed size per message when it's compressed,
        // because the uncompressed bytes are provided through an InputStream whose total size is
        // unknown until all bytes are read, and we don't know when it happens.
        byte[] stream = compressedFlag ? getCompressedBody() : getUncompressedBody();

        listener.onRawMessage(stream);

        // Done with this frame, begin processing the next header.
        state = GrpcDecodeState.HEADER;
        requiredLength = HEADER_LENGTH;
    }