spark源码-RPC通信机制

1.概述

​ 根据百度百科定义，RPC一般指远程过程调用，也是远程过程调用的缩写；

​ spark是一个分布式框架，分布式框架必然涉及节点与节点之间的通信；spark的不同组件节点之间也是通过RPC进行点对点通信；

​ 根据网上查到的资料了解到，spark参照akka在2.0之后自己实现了一套rpc通信逻辑，接下来就针对spark2.0后的rpc通信机制进行分析；

2.RPC通信原理

2.1 Actor体系

2.1.1.类的设计架构图

2.1.2.核心类分析

2.1.2.1.NettyRpcEnv-RPC通信环境

2.1.2.1.1.关联类

​ 在spark RPC通信机制中，RpcEnv是定义RPC环境的顶级抽象类，其唯一实现为NettyRpcEnv类；

​ 通过实例化NettyRpcEnv类完成对RpcEnv类的实例化；在实例化过程中，对NettyRpcEnv类中的属性进行初始化，完成RPC通信环境的准备；

2.1.2.1.2.属性初始化

​ 根据sparkConf中spark.rpc开头，网络io相关的配置，实例化TransportConf类；

​ 将该实例赋值给NettyRpcEnv的transportConf属性，完成属性初始化；

​ 配置中定义：

​ 每对RPC实体间建立的连接数量为1；

​ RPC服务端和客户端的线程数：numUsableCores参数 > 0时，用参数，否则在CPU可用数与8之间取小的一个；

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {

  //RPC通信配置：存储sparkConf中spark.rpc开头的进行网络io相关配置
  private[netty] val transportConf = SparkTransportConf.fromSparkConf(
    //根据sparkConf克隆一个新的conf对象
    //在conf对象中设置在每对RPC实体间建立的连接数量，默认是1
    conf.clone.set("spark.rpc.io.numConnectionsPerPeer", "1"),
    "rpc",
    //RPC服务端和客户端的线程数
    conf.getInt("spark.rpc.io.threads", numUsableCores))

  //直接实例化一个Dispatcher，根据numUsableCores设置消息分发器线程池线程数；
  private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)

  //处理ChunkFetchRequest和StreamRequest请求
  private val streamManager = new NettyStreamManager(this)

  //是一个创建TransportServer, TransportClientFactory，使用TransportChannelHandler建立netty channel pipeline的上下文
  private val transportContext = new TransportContext(transportConf,
    new NettyRpcHandler(dispatcher, this, streamManager))
  
  //创建传输客户端（TransportClient）的传输客户端工厂类
  private val clientFactory = transportContext.createClientFactory(createClientBootstraps())

  //TransportServer是RPC框架的服务端，可提供高效的、低级别的流服务
  @volatile private var server: TransportServer = _

  //[[RpcAddress]]和[[发件箱]]的映射：一个RpcEndpointRef对应一个发件箱，一个RpcEndpoint 可以有多个RpcEndpointRef
  //当我们连接到远程[[RpcAddress]]时，我们只是把消息放到它的[[Outbox]]来实现一个非阻塞的“发送”方法。
  private val outboxes = new ConcurrentHashMap[RpcAddress, Outbox]()
}

object SparkTransportConf {
  //根据sparkConf中的参数信息构建TransportConf；module = "rpc"；
  def fromSparkConf(_conf: SparkConf, module: String, numUsableCores: Int = 0): TransportConf = {
    //克隆新的sparkConf对象
    val conf = _conf.clone

    // 确定RPC服务端和客户端的线程数，并添加到配置中去作为客户端和服务端通信线程数
    val numThreads = defaultNumThreads(numUsableCores)
    conf.setIfMissing(s"spark.$module.io.serverThreads", numThreads.toString)
    conf.setIfMissing(s"spark.$module.io.clientThreads", numThreads.toString)

    //根据sparkConf实例化TransportConf，重新配置获取方法；
    new TransportConf(module, new ConfigProvider {
      override def get(name: String): String = conf.get(name)
      override def get(name: String, defaultValue: String): String = conf.get(name, defaultValue)
      override def getAll(): java.lang.Iterable[java.util.Map.Entry[String, String]] = {
        conf.getAll.toMap.asJava.entrySet()
      }
    })
  }
  
  // 确定RPC服务端和客户端的线程数
  private def defaultNumThreads(numUsableCores: Int): Int = {
    //参数 > 0时，用参数，否则在CPU可用数与8之间取小的一个
    val availableCores =
      if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
    math.min(availableCores, MAX_DEFAULT_NETTY_THREADS)
  }
}

//通信配置类：存储sparkConf中spark.rpc开头的进行网络io相关配置
public class TransportConf {
    private final String SPARK_NETWORK_IO_MODE_KEY;
    private final String SPARK_NETWORK_IO_PREFERDIRECTBUFS_KEY;
    private final String SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY;
    private final String SPARK_NETWORK_IO_BACKLOG_KEY;
    private final String SPARK_NETWORK_IO_NUMCONNECTIONSPERPEER_KEY;
    private final String SPARK_NETWORK_IO_SERVERTHREADS_KEY;
    private final String SPARK_NETWORK_IO_CLIENTTHREADS_KEY;
    private final String SPARK_NETWORK_IO_RECEIVEBUFFER_KEY;
    private final String SPARK_NETWORK_IO_SENDBUFFER_KEY;
    private final String SPARK_NETWORK_SASL_TIMEOUT_KEY;
    private final String SPARK_NETWORK_IO_MAXRETRIES_KEY;
    private final String SPARK_NETWORK_IO_RETRYWAIT_KEY;
    private final String SPARK_NETWORK_IO_LAZYFD_KEY;
    private final String SPARK_NETWORK_VERBOSE_METRICS;
    private final ConfigProvider conf;
    private final String module;

    public TransportConf(String module, ConfigProvider conf) {
        this.module = module;
        this.conf = conf;
        this.SPARK_NETWORK_IO_MODE_KEY = this.getConfKey("io.mode");
        this.SPARK_NETWORK_IO_PREFERDIRECTBUFS_KEY = this.getConfKey("io.preferDirectBufs");
        this.SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY = this.getConfKey("io.connectionTimeout");
        this.SPARK_NETWORK_IO_BACKLOG_KEY = this.getConfKey("io.backLog");
        this.SPARK_NETWORK_IO_NUMCONNECTIONSPERPEER_KEY = this.getConfKey("io.numConnectionsPerPeer");
        this.SPARK_NETWORK_IO_SERVERTHREADS_KEY = this.getConfKey("io.serverThreads");
        this.SPARK_NETWORK_IO_CLIENTTHREADS_KEY = this.getConfKey("io.clientThreads");
        this.SPARK_NETWORK_IO_RECEIVEBUFFER_KEY = this.getConfKey("io.receiveBuffer");
        this.SPARK_NETWORK_IO_SENDBUFFER_KEY = this.getConfKey("io.sendBuffer");
        this.SPARK_NETWORK_SASL_TIMEOUT_KEY = this.getConfKey("sasl.timeout");
        this.SPARK_NETWORK_IO_MAXRETRIES_KEY = this.getConfKey("io.maxRetries");
        this.SPARK_NETWORK_IO_RETRYWAIT_KEY = this.getConfKey("io.retryWait");
        this.SPARK_NETWORK_IO_LAZYFD_KEY = this.getConfKey("io.lazyFD");
        this.SPARK_NETWORK_VERBOSE_METRICS = this.getConfKey("io.enableVerboseMetrics");
    }

    private String getConfKey(String suffix) {
        return "spark." + this.module + "." + suffix;
    }
}

2.1.2.2.RpcEndpoint-RPC节点

2.1.2.2.1.关联类

​ RpcEndpoint类是RPC通信机制中所有节点类的顶级父类；

​ 如图所示，Master、Worker等都是rpc节点，受RpcEndpoint约束；

​ RpcEndpointRef是RpcEndpoint的引用，相当于RpcEndpoint的客户端；节点间通信都是通过调用对应节点的RpcEndpointRef对象实现；由RpcEndpointRef找到自己的RpcEndpoint将消息传输过去；

​ 一个RpcEndpoint可以有多个RpcEndpointRef；

​ NettyRpcEndpointRef是RpcEndpointRef的子类，重写了RpcEndpointRef中很多方法，在实例化RpcEndpointRef的时候都是实例化的NettyRpcEndpointRef对象；

2.1.2.2.2.发件节点获取收件节点的客户端RpcEndpointRef对象

​ 在发件节点中，根据收件节点URI调用NettyRpcEnv#asyncSetupEndpointRefByURI(uri: String)，异步获取收件节点引用对象；

private[spark] abstract class RpcEnv(conf: SparkConf) {
	def asyncSetupEndpointRefByURI(uri: String): Future[RpcEndpointRef]
}

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
  
  //根据节点URI异步节点引用对象
  def asyncSetupEndpointRefByURI(uri: String): Future[RpcEndpointRef] = {
    val addr = RpcEndpointAddress(uri)
    val endpointRef = new NettyRpcEndpointRef(conf, addr, this)
    val verifier = new NettyRpcEndpointRef(
      conf, RpcEndpointAddress(addr.rpcAddress, RpcEndpointVerifier.NAME), this)
    verifier.ask[Boolean](RpcEndpointVerifier.CheckExistence(endpointRef.name)).flatMap { find =>
      if (find) {
        Future.successful(endpointRef)
      } else {
        Future.failed(new RpcEndpointNotFoundException(uri))
      }
    }(ThreadUtils.sameThread)
  }
}

2.1.2.2.3.发件节点通过RpcEndpointRef向收件节点发送消息

发送节点通过调用收件节点引用对象RpcEndpointRef的send()、ask()方法进行消息发送；

private[spark] abstract class RpcEndpointRef(conf: SparkConf)
  extends Serializable with Logging {
    
  //发送单向异步消息；发送后不等的回信；
  //由接受节点的RpcEndpoint.receive()方法处理
  def send(message: Any): Unit

  //发送消息后在超时时间内等待回信；
  //由接受节点的RpcEndpoint.receiveAndReply()方法处理
  //只发送一次，不重复发送
  def ask[T: ClassTag](message: Any, timeout: RpcTimeout): Future[T]    
}

private[netty] class NettyRpcEndpointRef(
    @transient private val conf: SparkConf,
    private val endpointAddress: RpcEndpointAddress,
    @transient @volatile private var nettyEnv: NettyRpcEnv) extends RpcEndpointRef(conf) {
  
  override def ask[T: ClassTag](message: Any, timeout: RpcTimeout): Future[T] = {
    nettyEnv.ask(new RequestMessage(nettyEnv.address, this, message), timeout)
  }

  override def send(message: Any): Unit = {
    require(message != null, "Message is null")
    nettyEnv.send(new RequestMessage(nettyEnv.address, this, message))
  }
}

2.1.2.2.4.收件节点接受消息

private[spark] trait RpcEndpoint {

  //处理发件节点通过send方式发送过来的消息，不需要回信
  def receive: PartialFunction[Any, Unit] = {
    case _ => throw new SparkException(self + " does not implement 'receive'")
  }
  
  //处理发件节点通过ask方式发送过来的消息，需要回信
  def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
    case _ => context.sendFailure(new SparkException(self + " won't reply anything"))
  }
}

//具体处理逻辑在RpcEndpoint的子类中实现，不同用途的子类实现逻辑不一定一致；

2.1.2.2.5.消息从RpcEndpointRef到RpcEndpoint

如下图所示，消息发送节点（发件节点）将消息发送到消息接受节点（收件节点）流程：

​ 1、发件节点通过收件节点的引用NettyRpcEndpointRef对象（rpcEndRef）调用ask()方法进行消息发送；

​ 2、rpcEndRef.ask()通过rpcEndRef对象属性rpcEnv，调用rpcEnv.ask()进行处理；

​ 3、rpcEnv.ask()通过rpcE对象属性dispatcher，调用dispatcher.postLocalMessage()或dispatcher.postToOutbox()进行处理；

​ 4、针对发送消息到本地节点，由于每个节点在实例化的时候，都会向消息分发器Dispatcher进行注册，所以Dispatcher缓存了可以接受消息的节点信息–收件节点清单；在dispatcher.postLocalMessage()中，首先根据消息中的收件节点名称从收件节点清单中找到收件节点，然后向收件节点的收件箱中发送信息（调用Inbox.post()方法）；

​ 5、针对发送消息到远程节点，如果节点引用对象有维护通信客户端，通过通信客户端发送消息，否则，构建节点引用对象的发件箱，将消息发送发件箱；

至此，完成消息从发件节点到收件节点的流转；

private[netty] class NettyRpcEndpointRef(
    @transient private val conf: SparkConf,
    private val endpointAddress: RpcEndpointAddress,
    @transient @volatile private var nettyEnv: NettyRpcEnv) extends RpcEndpointRef(conf) {

  //向节点发送ask消息
  override def ask[T: ClassTag](message: Any, timeout: RpcTimeout): Future[T] = {
    //调用rpcEnv中的ask方法
    nettyEnv.ask(new RequestMessage(nettyEnv.address, this, message), timeout)
  }

  //向节点发送send消息
  override def send(message: Any): Unit = {
    require(message != null, "Message is null")
    //调用rpcEnv中的ask方法
    nettyEnv.send(new RequestMessage(nettyEnv.address, this, message))
  }
}

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
  
  private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)
  
  private[netty] def send(message: RequestMessage): Unit = {
    val remoteAddr = message.receiver.address
    if (remoteAddr == address) {//发送消息到本地节点
      // Message to a local RPC endpoint.
      try {
        
        //调用消息分发器postLocalMessage方法
        dispatcher.postOneWayMessage(message)
      } catch {
        case e: RpcEnvStoppedException => logDebug(e.getMessage)
      }
    } else {//通过RPC方式发送消息到远程节点
      // Message to a remote RPC endpoint.
      postToOutbox(message.receiver, OneWayOutboxMessage(message.serialize(this)))
    }
  }
  
  private[netty] def ask[T: ClassTag](message: RequestMessage, timeout: RpcTimeout): Future[T] = {
    //。。。。。。。其他代码

    try {
      if (remoteAddr == address) {//发送消息到本地节点
        //。。。。。。。其他代码
        
        //调用消息分发器postLocalMessage方法
        dispatcher.postLocalMessage(message, p)
      } else {//通过RPC方式发送消息到远程节点
        val rpcMessage = RpcOutboxMessage(message.serialize(this),
          onFailure,
          (client, response) => onSuccess(deserialize[Any](client, response)))
        postToOutbox(message.receiver, rpcMessage)
        promise.future.failed.foreach {
          case _: TimeoutException => rpcMessage.onTimeout()
          case _ =>
        }(ThreadUtils.sameThread)
      }
        
      //。。。。。。。其他代码
    } catch {
      case NonFatal(e) =>
        onFailure(e)
    }
   	//。。。。。。。其他代码
  }
  
  //发送消息到远程节点
  private def postToOutbox(receiver: NettyRpcEndpointRef, message: OutboxMessage): Unit = {
    //存在通信客户端，直接通过通信客户端发送消息
    if (receiver.client != null) {
      message.sendWith(receiver.client)
    } else {//不存在通信客户端
      require(receiver.address != null,
        "Cannot send message to client endpoint with no listen address.")
      //构建节点引用对象的发件箱
      val targetOutbox = {
        val outbox = outboxes.get(receiver.address)
        if (outbox == null) {
          val newOutbox = new Outbox(this, receiver.address)
          val oldOutbox = outboxes.putIfAbsent(receiver.address, newOutbox)
          if (oldOutbox == null) {
            newOutbox
          } else {
            oldOutbox
          }
        } else {
          outbox
        }
      }
      if (stopped.get) {
        // It's possible that we put `targetOutbox` after stopping. So we need to clean it.
        outboxes.remove(receiver.address)
        targetOutbox.stop()
      } else {
        //通过发件箱发送消息
        targetOutbox.send(message)
      }
    }
  }
}

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {
  //分发器中节点信息封装类，将同一个节点的名称、节点、节点引用、收件箱绑定到一起
  private class EndpointData(
      val name: String,
      val endpoint: RpcEndpoint,
      val ref: NettyRpcEndpointRef) {
    val inbox = new Inbox(ref, endpoint)
  }
  
  //节点名称和节点信息封装类的映射关系：可以根据节点名称找到节点信息封装类
  private val endpoints: ConcurrentMap[String, EndpointData] =
    new ConcurrentHashMap[String, EndpointData]
  //分发器中注册的收件节点（收件人）
  private val receivers = new LinkedBlockingQueue[EndpointData]
  
  //发送消息到本地节点
  def postLocalMessage(message: RequestMessage, p: Promise[Any]): Unit = {
    val rpcCallContext =
      new LocalNettyRpcCallContext(message.senderAddress, p)
    //封装消息
    val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
    //从消息中获取收件节点名称，发送消息
    postMessage(message.receiver.name, rpcMessage, (e) => p.tryFailure(e))
  }
  
  def postOneWayMessage(message: RequestMessage): Unit = {
    //从消息中获取收件节点名称，封装消息，发送消息
    postMessage(message.receiver.name, OneWayMessage(message.senderAddress, message.content),
      (e) => throw e)
  }
  
  //发送消息
  private def postMessage(
      endpointName: String,
      message: InboxMessage,
      callbackIfStopped: (Exception) => Unit): Unit = {
    val error = synchronized {
      //根据收件节点名称从分发器缓存中找到收件节点对象
      val data = endpoints.get(endpointName)
      if (stopped) {
        Some(new RpcEnvStoppedException())
      } else if (data == null) {
        Some(new SparkException(s"Could not find $endpointName."))
      } else {
        //向收件节点的收件箱中发送信息
        data.inbox.post(message)
        receivers.offer(data)
        None
      }
    }
    // We don't need to call `onStop` in the `synchronized` block
    error.foreach(callbackIfStopped)
  }
}

2.1.2.3.Dispatcher-消息分发器

如下图所示：

​ 在NettyRpcEnv中有属性dispatcher，该属性通过new Dispatcher实现初始化；即在RpcEnv实例化过程中，完成了Dispatcher的实例化过程；

​ Dispatcher类中存在2个内部类：EndpointData类和MessageLoop类；EndpointData类用于封装通信节点信息，包括节点、节点引用、节点名称、节点收件箱；MessageLoop类中描述了一个线程分发消息到节点进行消息处理的逻辑；

2.1.2.2.1.Dispather内部类

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {

  //通信节点信息类：将同一个节点的名字、节点对象、节点引用对象、节点收件箱绑定到一起
  private class EndpointData(
      val name: String,
      val endpoint: RpcEndpoint,
      val ref: NettyRpcEndpointRef) {
    val inbox = new Inbox(ref, endpoint)
  }
    
  //消息循环类线程：通过先进先出的方式从收件节点阻塞队列（节点注册时缓存的节点数据）中获取收件节点，调用收件节点处理消息的能力完成消息处理
  private class MessageLoop extends Runnable {
    override def run(): Unit = {
      try {
        while (true) {
          try {
            //通过先进先出的方式从收件节点阻塞队列（节点注册时缓存的节点数据）中获取收件节点
            val data = receivers.take()
            if (data == PoisonPill) {
              // Put PoisonPill back so that other MessageLoops can see it.
              receivers.offer(PoisonPill)
              return
            }
            //调用收件节点处理消息的能力完成消息处理
            data.inbox.process(Dispatcher.this)
          } catch {
            case NonFatal(e) => logError(e.getMessage, e)
          }
        }
      } catch {
        case _: InterruptedException => // exit
        case t: Throwable =>
          try {
            // Re-submit a MessageLoop so that Dispatcher will still work if
            // UncaughtExceptionHandler decides to not kill JVM.
            threadpool.execute(new MessageLoop)
          } finally {
            throw t
          }
      }
    }
  }
}

2.1.2.2.2.Dispather初始化

​ 消息分发器通过ConcurrentHashMap、LinkedBlockingQueue缓存通信节点的注册信息；其中LinkedBlockingQueue节点缓存所有可用通信节点的节点信息封装对象；这些注册信息在通信节点向dispatcher注册时完成信息填充；

​ 消息分发处理的线程池在Dispatcher实例化时完成初始化，线程池通过执行消息循环线程完成消息分发处理；线程池中线程数确定方式：参数配置中spark.rpc.netty.dispatcher.numThreads参数的值有设置，就用设置的参数值，否则在2和可用cpu核数中取大；

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {

  //节点名称-节点信息封装对象的映射，通信节点向dispatcher注册时进行信息填充
  private val endpoints: ConcurrentMap[String, EndpointData] =
    new ConcurrentHashMap[String, EndpointData]
  
  //节点对象-节点引用对象的映射，通信节点向dispatcher注册时进行信息填充
  private val endpointRefs: ConcurrentMap[RpcEndpoint, RpcEndpointRef] =
    new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]

  //收件节点队列，一个阻塞队列：通信节点向dispatcher注册时进行信息填充
  private val receivers = new LinkedBlockingQueue[EndpointData]

  //如果调度程序已停止，则为true。一旦停止，所有发布的消息将立即被退回。
  @GuardedBy("this")
  private var stopped = false
    
  //消息分发处理的线程池
  private val threadpool: ThreadPoolExecutor = {
    //可以cpu核数：参数>0用参数值，否则使用jvm可以进程数
    val availableCores =
      if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
    //确定线程池中线程数：参数配置中spark.rpc.netty.dispatcher.numThreads参数的值有设置，就用设置的参数值，否则在2和可用cpu核数中取大
    val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
      math.max(2, availableCores))
      
    //根据线程数初始化线程池
    val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
    for (i <- 0 until numThreads) {
      //执行消息循环线程，完成消息分发处理
      pool.execute(new MessageLoop)
    }
    pool
  }
 
  //指示MessageLoop应该退出其消息循环的有害端点。
  private val PoisonPill = new EndpointData(null, null, null)
}

2.1.2.2.3.向dispatcher中注册通信节点

​ 节点注册主要完成如下事件：

​ 在disaptcher中缓存节点名称-节点信息封装对象的映射，ConcurrentHashMap缓存；

​ 在disaptcher中缓存节点对象-节点引用对象的映射，ConcurrentHashMap缓存；

​ 在disaptcher中缓存收件节点，LinkedBlockingQueue缓存；

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {

  private class EndpointData(
      val name: String,
      val endpoint: RpcEndpoint,
      val ref: NettyRpcEndpointRef) {
    val inbox = new Inbox(ref, endpoint)
  }

  private val endpoints: ConcurrentMap[String, EndpointData] =
    new ConcurrentHashMap[String, EndpointData]
  private val endpointRefs: ConcurrentMap[RpcEndpoint, RpcEndpointRef] =
    new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]
  private val receivers = new LinkedBlockingQueue[EndpointData]

  @GuardedBy("this")
  private var stopped = false

  //向dispatcher中注册节点信息
  def registerRpcEndpoint(name: String, endpoint: RpcEndpoint): NettyRpcEndpointRef = {
    val addr = RpcEndpointAddress(nettyEnv.address, name)
    val endpointRef = new NettyRpcEndpointRef(nettyEnv.conf, addr, nettyEnv)
    synchronized {
      if (stopped) {
        throw new IllegalStateException("RpcEnv has been stopped")
      }
      //封装节点信息，完成节点名称-节点信息封装对象的映射缓存
      if (endpoints.putIfAbsent(name, new EndpointData(name, endpoint, endpointRef)) != null) {
        throw new IllegalArgumentException(s"There is already an RpcEndpoint called $name")
      }
      //完成节点对象-节点引用对象的映射缓存
      val data = endpoints.get(name)
      endpointRefs.put(data.endpoint, data.ref)
      //节点阻塞队列缓存
      receivers.offer(data)  // for the OnStart message
    }
    
    //返回节点引用对象
    endpointRef
  }
}

2.1.2.2.4.通过dispatcher进行消息发送

​ 通过将消息发送到收件节点的收件箱完成消息发送；

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {
   
  //向所有节点发送消息
  def postToAll(message: InboxMessage): Unit = {
    val iter = endpoints.keySet().iterator()
    while (iter.hasNext) {
      val name = iter.next
        postMessage(name, message, (e) => { e match {
          case e: RpcEnvStoppedException => logDebug (s"Message $message dropped. ${e.getMessage}")
          case e: Throwable => logWarning(s"Message $message dropped. ${e.getMessage}")
        }}
      )}
  }

  //发布远程节点发送过来的消息
  def postRemoteMessage(message: RequestMessage, callback: RpcResponseCallback): Unit = {
    //封装发送内容
    val rpcCallContext =
      new RemoteNettyRpcCallContext(nettyEnv, callback, message.senderAddress)
    //封装消息对象
    val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
    //发送消息
    postMessage(message.receiver.name, rpcMessage, (e) => callback.onFailure(e))
  }

  //发布本地节点发送过来的消息
  def postLocalMessage(message: RequestMessage, p: Promise[Any]): Unit = {
    //封装发送内容
    val rpcCallContext =
      new LocalNettyRpcCallContext(message.senderAddress, p)
    //封装消息对象
    val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
    //发送消息
    postMessage(message.receiver.name, rpcMessage, (e) => p.tryFailure(e))
  }

  //发送单向无返回消息
  def postOneWayMessage(message: RequestMessage): Unit = {
    //封装消息对象、发送消息
    postMessage(message.receiver.name, OneWayMessage(message.senderAddress, message.content),
      (e) => throw e)
  }

  /**
   * 发送消息到对应节点
   *
   * @param endpointName 节点名称
   * @param message 需要发送的消息
   * @param callbackIfStopped 消息发送结束后执行的回调函数
   */
  private def postMessage(
      endpointName: String,
      message: InboxMessage,
      callbackIfStopped: (Exception) => Unit): Unit = {
    val error = synchronized {
      //根据节点名称从缓存中获取对于节点信息对象
      val data = endpoints.get(endpointName)
      if (stopped) {
        Some(new RpcEnvStoppedException())
      } else if (data == null) {
        Some(new SparkException(s"Could not find $endpointName."))
      } else {
        //向节点的收件箱发送消息
        data.inbox.post(message)
        //将节点信息对象插入队列尾部
        receivers.offer(data)
        None
      }
    }
    // 消息发送完成，执行回调函数
    error.foreach(callbackIfStopped)
  }
}

2.1.2.2.5.dispatcher分发处理消息

​ 通过线程池执行消息循环线程，在消息循环线程中，从阻塞队列中获取收件节点信息，然后向收件节点的收件箱发送消息；当收件节点的阻塞队列没有数据时，线程阻塞，知道由新的节点进入队列，线程不在阻塞，继续获取节点先后执行；

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {

  def verify(name: String): Boolean = {
    endpoints.containsKey(name)
  }

  //dispatcher实例化时完成线程池初始化；线程池执行消息循环线程
  private val threadpool: ThreadPoolExecutor = {
    val availableCores =
      if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
    val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
      math.max(2, availableCores))
    val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
    for (i <- 0 until numThreads) {
      pool.execute(new MessageLoop)
    }
    pool
  }

  //消息循环类线程：通过先进先出的方式从收件节点阻塞队列（节点注册时缓存的节点数据）中获取收件节点，调用收件节点处理消息的能力完成消息处理
  private class MessageLoop extends Runnable {
    override def run(): Unit = {
      try {
        while (true) {
          try {
            val data = receivers.take()
            if (data == PoisonPill) {
              // Put PoisonPill back so that other MessageLoops can see it.
              receivers.offer(PoisonPill)
              return
            }
            data.inbox.process(Dispatcher.this)
          } catch {
            case NonFatal(e) => logError(e.getMessage, e)
          }
        }
      } catch {
        case _: InterruptedException => // exit
        case t: Throwable =>
          try {
            // Re-submit a MessageLoop so that Dispatcher will still work if
            // UncaughtExceptionHandler decides to not kill JVM.
            threadpool.execute(new MessageLoop)
          } finally {
            throw t
          }
      }
    }
  }

  /** A poison endpoint that indicates MessageLoop should exit its message loop. */
  private val PoisonPill = new EndpointData(null, null, null)
}

2.1.2.2.6.总结

​ 1、一个RpcEnv（RPC通信环境）绑定一个消息分发器；

​ 2、通信节点实例化时，需要向消息分发器注册，消息分发器中缓存所有可用通信节点信息；

​ 3、消息分发器提供节点注册、反注册、发送消息到收件节点收件箱、分发消息到收件节点进行消息处理的能力；

​ 4、消息分发器就是一个通信中介，发件节点将消息发送给消息分发器，消息分发器将消息发送到收件节点的收件箱；

​ 5、消息分发器维护了一个线程池，池中允许多个消息循环线程，通过先进先出的方式从收件节点阻塞队列（节点注册时缓存的节点数据）中获取收件节点，调用收件节点处理消息的能力完成消息处理；

2.1.2.4.Inbox-收件箱

​ 如下图所示：收件箱Inbox持有节点对象、节点引用对象属性，可以标识收件箱归属节点；节点通过messages属性对收件箱中消息进行缓存，messages时一个InboxMessage消息的列表；

​ 在Inbox中，提供对消息的缓存、调用节点能力处理消息的能力；这2个能力都需要通过dispatcher调用才能生效；

2.1.2.4.1.实例化Inbox

在实例化Inbox过程中，完成消息缓存队列的初始化，已经向消息缓存队列中添加Onstart消息；

每个节点向消息分发器注册，都会绑定一个收件箱并向收件箱中添加第一条消息-Onstart消息；该消息在节点注册完成后，由消息分发器线程池拉起注册节点的消息循环线程执行消息分发，最后由注册节点的onstart方法处理；

private[netty] class Inbox(
    val endpointRef: NettyRpcEndpointRef,
    val endpoint: RpcEndpoint)
  extends Logging {

  inbox =>  // Give this an alias so we can use it more clearly in closures.

  //初始化消息缓存队列
  @GuardedBy("this")
  protected val messages = new java.util.LinkedList[InboxMessage]()

  /** True if the inbox (and its associated endpoint) is stopped. */
  @GuardedBy("this")
  private var stopped = false

  /** Allow multiple threads to process messages at the same time. */
  @GuardedBy("this")
  private var enableConcurrent = false

  /** The number of threads processing messages for this inbox. */
  @GuardedBy("this")
  private var numActiveThreads = 0

  //将Onstart消息第一个添加到收件箱的消息缓存队列
  //Onstart消息在节点实例化后，向消息分发器dispatcher注册后执行（完成注册过程中，已经将收件箱绑定到节点并添加Onstart消息），由消息分发器的线程池自动拉起注册节点的消息循环线程执行Onstart消息处理；
  inbox.synchronized {
    messages.add(OnStart)
  }
}

2.1.2.4.2.缓存消息

通过LinkedList提供消息缓存能力；

当收件箱接收到通过dispatcher发送过来的消息后，即缓存到消息列表中；

private[netty] class Inbox(
    val endpointRef: NettyRpcEndpointRef,
    val endpoint: RpcEndpoint)
  extends Logging {

  inbox =>  // Give this an alias so we can use it more clearly in closures.

  //提供缓存消息能力
  @GuardedBy("this")
  protected val messages = new java.util.LinkedList[InboxMessage]()
      
  def post(message: InboxMessage): Unit = inbox.synchronized {
    if (stopped) {
      // We already put "OnStop" into "messages", so we should drop further messages
      onDrop(message)
    } else {
      //将新的消息添加到消息缓存列表中
      messages.add(message)
      false
    }
  }
      
  protected def onDrop(message: InboxMessage): Unit = {
    logWarning(s"Drop $message because $endpointRef is stopped")
  }
}

2.1.2.4.3.处理消息

收件箱中消息的处理，根据先进先出规则，从前向后，一条一条处理；

每一条消息都有一个单独线程处理；

消息的处理，都有收件箱所属节点的对应方法处理；

消息类型与节点中处理消息的方法的对应关系：

​ RpcMessage—>receiveAndReply

​ OneWayMessage—>receiveAndReply

​ OnStart—>onStart

​ OnStop—>onStop

​ RemoteProcessConnected—>onConnected

​ RemoteProcessDisconnected—>onDisconnected

​ RemoteProcessConnectionError—>onNetworkError

处理OnStart消息后，打开多线程处理消息的开关，允许多个线程同时处理收件箱中消息；

处理OnStop消息后，从消息分发器的收件节点列表中移除执行OnStop消息的节点；

private[netty] class Inbox(
    val endpointRef: NettyRpcEndpointRef,
    val endpoint: RpcEndpoint)
  extends Logging {

  inbox =>  // Give this an alias so we can use it more clearly in closures.

  //提供缓存消息能力
  @GuardedBy("this")
  protected val messages = new java.util.LinkedList[InboxMessage]()
      
  def process(dispatcher: Dispatcher): Unit = {
    var message: InboxMessage = null
    inbox.synchronized {
      if (!enableConcurrent && numActiveThreads != 0) {
        return
      }
      
      //每次处理消息缓存中的头部消息，即消息缓存中第一条消息
      message = messages.poll()
      if (message != null) {
        //每条消息由一个单独线程处理，没处理一天消息，收件箱处理消息的线程数+1
        numActiveThreads += 1
      } else {
        return
      }
    }
     
    //所有的消息，都由收件箱所属节点的对应方法处理
    while (true) {
      safelyCall(endpoint) {
        message match {
          case RpcMessage(_sender, content, context) =>
            try {
              endpoint.receiveAndReply(context).applyOrElse[Any, Unit](content, { msg =>
                throw new SparkException(s"Unsupported message $message from ${_sender}")
              })
            } catch {
              case e: Throwable =>
                context.sendFailure(e)
                // Throw the exception -- this exception will be caught by the safelyCall function.
                // The endpoint's onError function will be called.
                throw e
            }

          case OneWayMessage(_sender, content) =>
            endpoint.receive.applyOrElse[Any, Unit](content, { msg =>
              throw new SparkException(s"Unsupported message $message from ${_sender}")
            })

          case OnStart =>
            endpoint.onStart()
            //允许多个线程同时处理收件箱中消息
            if (!endpoint.isInstanceOf[ThreadSafeRpcEndpoint]) {
              inbox.synchronized {
                if (!stopped) {
                  enableConcurrent = true
                }
              }
            }

          case OnStop =>
            val activeThreads = inbox.synchronized { inbox.numActiveThreads }
            assert(activeThreads == 1,
              s"There should be only a single active thread but found $activeThreads threads.")
            //从消息分发器的收件节点列表中移除当前节点
            dispatcher.removeRpcEndpointRef(endpoint)
            endpoint.onStop()
            assert(isEmpty, "OnStop should be the last message")

          case RemoteProcessConnected(remoteAddress) =>
            endpoint.onConnected(remoteAddress)

          case RemoteProcessDisconnected(remoteAddress) =>
            endpoint.onDisconnected(remoteAddress)

          case RemoteProcessConnectionError(cause, remoteAddress) =>
            endpoint.onNetworkError(cause, remoteAddress)
        }
      }

      inbox.synchronized {
        // "enableConcurrent" will be set to false after `onStop` is called, so we should check it
        // every time.
        if (!enableConcurrent && numActiveThreads != 1) {
          // If we are not the only one worker, exit
          numActiveThreads -= 1
          return
        }
        message = messages.poll()
        if (message == null) {
          numActiveThreads -= 1
          return
        }
      }
    }
  }
}

2.1.2.5.Outbox-发件箱

在发件箱中，通过LinkedList列表缓存OutboxMessage消息；并在发件箱中维护通信客户端TransportClient对象；提供消息发送send能力；

只有处理发送到远端其他节点（非本节点）的消息，才会用到发件箱；通过发件箱的通信客户端维护与远端地址的连接，最后调用OutboxMessage.sendWith()方法处理消息；

2.1.2.5.1.实例化Outbox

在节点引用对象没有维护通信客户端属性时，实例化发件箱并绑定到节点引用中；

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
    
  private def postToOutbox(receiver: NettyRpcEndpointRef, message: OutboxMessage): Unit = {
    if (receiver.client != null) {
      message.sendWith(receiver.client)
    } else {
      //当节点引用对象没有维护通信客户端，需要实例化发件箱，并将发件箱信息缓存在rpcEnv中
      require(receiver.address != null,
        "Cannot send message to client endpoint with no listen address.")
      val targetOutbox = {
        val outbox = outboxes.get(receiver.address)
        if (outbox == null) {
          val newOutbox = new Outbox(this, receiver.address)
          val oldOutbox = outboxes.putIfAbsent(receiver.address, newOutbox)
          if (oldOutbox == null) {
            newOutbox
          } else {
            oldOutbox
          }
        } else {
          outbox
        }
      }
      if (stopped.get) {
        // It's possible that we put `targetOutbox` after stopping. So we need to clean it.
        outboxes.remove(receiver.address)
        targetOutbox.stop()
      } else {
        targetOutbox.send(message)
      }
    }
  }
}

2.1.2.5.2.消息发送

首先，将消息添加到发件箱消息缓存队列；

其次，保证发件箱的通信客户端不为空，否则根据远端地址创建通信客户端；

然后，根据先进先出规则，循环处理消息队列所有消息（OutboxMessage.sendWith()方法处理消息），直至消息清空，结束消息处理；

private[netty] class Outbox(nettyEnv: NettyRpcEnv, val address: RpcAddress) {

  outbox => // Give this an alias so we can use it more clearly in closures.
  
  //消息队列
  @GuardedBy("this")
  private val messages = new java.util.LinkedList[OutboxMessage]
  @GuardedBy("this")
  private var client: TransportClient = null
    
  //消息发件箱的发送能力
  def send(message: OutboxMessage): Unit = {
    val dropped = synchronized {
      if (stopped) {
        true
      } else {
        //将消息缓存到发件箱的消息缓存列表
        messages.add(message)
        false
      }
    }
    if (dropped) {
      message.onFailure(new SparkException("Message is dropped because Outbox is stopped"))
    } else {
      //清空消息队列：将消息缓存队列中所有消息都发送出去
      drainOutbox()
    }
  }
  
  //清空消息队列
  private def drainOutbox(): Unit = {
    var message: OutboxMessage = null
    synchronized {
      if (stopped) {
        return
      }
      if (connectFuture != null) {
        // 正在创建连接远端的通信客户端
        return
      }
      if (client == null) {
        // 缺少通信客户端，启动连接任务
        launchConnectTask()
        return
      }
      if (draining) {
        // 已经存在线程正在执行清空消息队列任务，则不在执行本次清空任务
        return
      }
      
      //从对头获取本次处理从消息
      message = messages.poll()
      if (message == null) {
        return
      }
      //标识已有任务正在执行清空任务
      draining = true
    }
    while (true) {//循环处理消息队列中所有消息，直至清空消息队列
      try {
        val _client = synchronized { client }
        if (_client != null) {
          //客户端连接建立成功，发送消息
          message.sendWith(_client)
        } else {
          assert(stopped == true)
        }
      } catch {
        case NonFatal(e) =>
          handleNetworkFailure(e)
          return
      }
      synchronized {
        if (stopped) {
          return
        }
        message = messages.poll()
        if (message == null) {//消息队列清空，不在循环
          draining = false
          return
        }
      }
    }
  }

  private def launchConnectTask(): Unit = {
    //由rpcEnv中的客户端连接池负债执行创建通信客户端的线程
    connectFuture = nettyEnv.clientConnectionExecutor.submit(new Callable[Unit] {

      override def call(): Unit = {
        try {
          //根据远端地址建立连接远端的通信客户端
          val _client = nettyEnv.createClient(address)
          outbox.synchronized {
            //通信客户端绑定到发件箱
            client = _client
            if (stopped) {
              closeClient()
            }
          }
        } catch {
          case ie: InterruptedException =>
            // exit
            return
          case NonFatal(e) =>
            outbox.synchronized { connectFuture = null }
            handleNetworkFailure(e)
            return
        }
        //标识与远端连接建立成功
        outbox.synchronized { connectFuture = null }
        // 通信客户端建立成功，开始清空消息列表
        drainOutbox()
      }
    })
  }
}

2.1.2.5.3.OutboxMessage

OutboxMessage是发件箱消息的顶级trait，定义了消息发送和失败处理的行为；有2个子类，OneWayOutboxMessag消息不需要返回信息，调用通信客户端的send方法完成消息发送；RpcOutboxMessage消息有返回值，调用通信客户端sendRpc方法完成消息发送；

//发件箱消息的顶级trait：定义消息发送和失败处理行为
private[netty] sealed trait OutboxMessage {

  def sendWith(client: TransportClient): Unit

  def onFailure(e: Throwable): Unit

}

//单向无返回的发件箱消息封装类
private[netty] case class OneWayOutboxMessage(content: ByteBuffer) extends OutboxMessage
  with Logging {

  override def sendWith(client: TransportClient): Unit = {
    client.send(content)
  }

  override def onFailure(e: Throwable): Unit = {
    e match {
      case e1: RpcEnvStoppedException => logDebug(e1.getMessage)
      case e1: Throwable => logWarning(s"Failed to send one-way RPC.", e1)
    }
  }

}

//需要RPC方式发送的发件箱消息封装类
private[netty] case class RpcOutboxMessage(
    content: ByteBuffer,
    _onFailure: (Throwable) => Unit,
    _onSuccess: (TransportClient, ByteBuffer) => Unit)
  extends OutboxMessage with RpcResponseCallback with Logging {

  private var client: TransportClient = _
  private var requestId: Long = _

  override def sendWith(client: TransportClient): Unit = {
    this.client = client
    this.requestId = client.sendRpc(content, this)
  }

  def onTimeout(): Unit = {
    if (client != null) {
      client.removeRpcRequest(requestId)
    } else {
      logError("Ask timeout before connecting successfully")
    }
  }

  override def onFailure(e: Throwable): Unit = {
    _onFailure(e)
  }

  override def onSuccess(response: ByteBuffer): Unit = {
    _onSuccess(client, response)
  }

}

2.2 Transport体系

client、server通信模型：在初始化管道时定义；

2.2.1.实例化TransportContext

在实例化RpcEnv过程中，对transportContext属性进行初始化，指定rpcHandler为NettyRpcHandler；

在NettyRpcHandler上绑定dispatcher、streamManager

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
  //实例化RpcEnv过程中，对transportContext属性进行初始化，指定rpcHandler为NettyRpcHandler，在NettyRpcHandler上绑定dispatcher、streamManager
  private val transportContext = new TransportContext(transportConf,
    new NettyRpcHandler(dispatcher, this, streamManager))
}

public class TransportContext {
  
  private final TransportConf conf;
  private final RpcHandler rpcHandler;
  private final boolean closeIdleConnections;
  
  private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
  private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
  
  public TransportContext(TransportConf conf, RpcHandler rpcHandler) {
    this(conf, rpcHandler, false);
  }
  
  //构造函数，完成属性绑定
  public TransportContext(
      TransportConf conf,
      RpcHandler rpcHandler,
      boolean closeIdleConnections) {
    this.conf = conf;
    this.rpcHandler = rpcHandler;
    this.closeIdleConnections = closeIdleConnections;
  }
}

2.2.1.实例化TransportServer

2.2.1.1.入口

TransportServer的实例化，在创建driver端执行环境，初始化sparkEnv的过程中完成；通过TransportContext#createServer()完成server的创建；

SparkEnv#createDriverEnv() —> SparkEnv#create() —> RpcEnv#create() —> NettyRpcEnvFactory#create()

—> NettyRpcEnv#startServer() —> TransportContext#createServer()

//构建sparkEnv过程中，初始化rpcEnv
object SparkEnv extends Logging {
  private def create(
      conf: SparkConf,
      executorId: String,
      bindAddress: String,
      advertiseAddress: String,
      port: Option[Int],
      isLocal: Boolean,
      numUsableCores: Int,
      ioEncryptionKey: Option[Array[Byte]],
      listenerBus: LiveListenerBus = null,
      mockOutputCommitCoordinator: Option[OutputCommitCoordinator] = None): SparkEnv = {
    val rpcEnv = RpcEnv.create(systemName, bindAddress, advertiseAddress, port.getOrElse(-1), conf,
      securityManager, numUsableCores, !isDriver)
  }
}

private[spark] object RpcEnv {
  def create(
      name: String,
      bindAddress: String,
      advertiseAddress: String,
      port: Int,
      conf: SparkConf,
      securityManager: SecurityManager,
      numUsableCores: Int,
      clientMode: Boolean): RpcEnv = {
    val config = RpcEnvConfig(conf, name, bindAddress, advertiseAddress, port, securityManager,
      numUsableCores, clientMode)
    //通过rpcEnv工厂创建rpcEnv
    new NettyRpcEnvFactory().create(config)
  }
}

private[rpc] class NettyRpcEnvFactory extends RpcEnvFactory with Logging {
  def create(config: RpcEnvConfig): RpcEnv = {
    val sparkConf = config.conf
    // Use JavaSerializerInstance in multiple threads is safe. However, if we plan to support
    // KryoSerializer in future, we have to use ThreadLocal to store SerializerInstance
    val javaSerializerInstance =
      new JavaSerializer(sparkConf).newInstance().asInstanceOf[JavaSerializerInstance]
    val nettyEnv =
      new NettyRpcEnv(sparkConf, javaSerializerInstance, config.advertiseAddress,
        config.securityManager, config.numUsableCores)
    if (!config.clientMode) {
      val startNettyRpcEnv: Int => (NettyRpcEnv, Int) = { actualPort =>
        //创建RpcEnv过程中，针对非clientMode，启动TransportServer节点
        nettyEnv.startServer(config.bindAddress, actualPort)
        (nettyEnv, nettyEnv.address.port)
      }
      try {
        Utils.startServiceOnPort(config.port, startNettyRpcEnv, sparkConf, config.name)._1
      } catch {
        case NonFatal(e) =>
          nettyEnv.shutdown()
          throw e
      }
    }
    nettyEnv
  }
}

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
  
  def startServer(bindAddress: String, port: Int): Unit = {
    val bootstraps: java.util.List[TransportServerBootstrap] =
      if (securityManager.isAuthenticationEnabled()) {
        java.util.Arrays.asList(new AuthServerBootstrap(transportConf, securityManager))
      } else {
        java.util.Collections.emptyList()
      }
    //创建TransportServer节点
    server = transportContext.createServer(bindAddress, port, bootstraps)
    //将节点注册到消息分发器中
    dispatcher.registerRpcEndpoint(
      RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))
  }
}

2.2.1.2.实例化TransportServer逻辑

在创建TransportServer节点过程中，创建Netty的服务端根引导程序，在跟引导程序上设置置管道初始化回调函数、绑定Socket的监听端口；

初始化管道时，创建TransportChannelHandler，将将TransportChannelHandler添加到socketChannel中；

创建TransportChannelHandler时，将创建TransportResponseHandler、TransportRequestHandler、TransportClient并绑定到TransportChannelHandler；

SocketChannel中消息/数据处理流程：

public class TransportContext {
  
  //在TransportContext实例化时，指定rpcHandler为NettyRpcHandler，在NettyRpcHandler上绑定dispatcher、streamManager
  private final RpcHandler rpcHandler;
  
  public TransportServer createServer(
      String host, int port, List<TransportServerBootstrap> bootstraps) {
    //构建server
    return new TransportServer(this, host, port, rpcHandler, bootstraps);
  }
  
  //初始化管道
  public TransportChannelHandler initializePipeline(
      SocketChannel channel,
      RpcHandler channelRpcHandler) {
    try {
      //创建channelhandler
      TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
      
      //定义管道中消息/数据处理流程
      channel.pipeline()
        .addLast("encoder", ENCODER)
        .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
        .addLast("decoder", DECODER)
        .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
      	//将channelHandler添加到socketChannel中
        .addLast("handler", channelHandler);
      return channelHandler;
    } catch (RuntimeException e) {
      logger.error("Error while initializing Netty pipeline", e);
      throw e;
    }
  }
  
  private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
    //创建TransportResponseHandler
    TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
    //创建TransportClient
    TransportClient client = new TransportClient(channel, responseHandler);
    //创建TransportRequestHandler
    TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
      rpcHandler, conf.maxChunksBeingTransferred());
    //将TransportResponseHandler、TransportRequestHandler、TransportClient绑定到TransportChannelHandler
    return new TransportChannelHandler(client, responseHandler, requestHandler,
      conf.connectionTimeoutMs(), closeIdleConnections);
  }
}

public class TransportServer implements Closeable {
  public TransportServer(
      TransportContext context,
      String hostToBind,
      int portToBind,
      RpcHandler appRpcHandler,
      List<TransportServerBootstrap> bootstraps) {
    //属性绑定
    this.context = context;
    this.conf = context.getConf();
    this.appRpcHandler = appRpcHandler;
    this.bootstraps = Lists.newArrayList(Preconditions.checkNotNull(bootstraps));

    boolean shouldClose = true;
    try {
      //初始化server设置
      init(hostToBind, portToBind);
      shouldClose = false;
    } finally {
      if (shouldClose) {
        JavaUtils.closeQuietly(this);
      }
    }
  }
  
  private void init(String hostToBind, int portToBind) {
		//根据Netty的API文档，Netty服务端需同时创建bossGroup和workerGroup
    IOMode ioMode = IOMode.valueOf(conf.ioMode());
    EventLoopGroup bossGroup = NettyUtils.createEventLoop(ioMode, 1,
      conf.getModuleName() + "-boss");
    EventLoopGroup workerGroup =  NettyUtils.createEventLoop(ioMode, conf.serverThreads(),
      conf.getModuleName() + "-server");
		// 创建一个汇集ByteBuf但对本地线程缓存禁用的分配器
    PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
      conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());

    // 创建Netty的服务端根引导程序并对其进行配置
    bootstrap = new ServerBootstrap()
      .group(bossGroup, workerGroup)
      .channel(NettyUtils.getServerChannelClass(ioMode))
      .option(ChannelOption.ALLOCATOR, allocator)
      .option(ChannelOption.SO_REUSEADDR, !SystemUtils.IS_OS_WINDOWS)
      .childOption(ChannelOption.ALLOCATOR, allocator);

    this.metrics = new NettyMemoryMetrics(
      allocator, conf.getModuleName() + "-server", conf);

    if (conf.backLog() > 0) {
      bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
    }

    if (conf.receiveBuf() > 0) {
      bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
    }

    if (conf.sendBuf() > 0) {
      bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
    }
		
    // 为根引导程序设置管道初始化回调函数
    bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
      @Override
      protected void initChannel(SocketChannel ch) {
        RpcHandler rpcHandler = appRpcHandler;
        for (TransportServerBootstrap bootstrap : bootstraps) {
          rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
        }
        //初始化管道
        context.initializePipeline(ch, rpcHandler);
      }
    });

    // 给根引导程序绑定Socket的监听端口
    InetSocketAddress address = hostToBind == null ?
        new InetSocketAddress(portToBind): new InetSocketAddress(hostToBind, portToBind);
    channelFuture = bootstrap.bind(address);
    channelFuture.syncUninterruptibly();

    port = ((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
    logger.debug("Shuffle server started on port: {}", port);
  }
}

2.2.2.实例化TransportClient

2.2.2.1.入口

入口一：

​ Outbox#send() —> Outbox#drainOutbox() —> Outbox#launchConnectTask() —> NettyRpcEnv#createClient() —> TransportClientFactory#createClient();

​ 在发件箱清空消息队列时，如果发件箱没有绑定通信客户端client，调用NettyRpcEnv#createClient()方法，最终通过TransportClient工厂对象来实例化一个TransportClient对象；

private[netty] class Outbox(nettyEnv: NettyRpcEnv, val address: RpcAddress) {

  outbox => // Give this an alias so we can use it more clearly in closures.
  
  //消息队列
  @GuardedBy("this")
  private val messages = new java.util.LinkedList[OutboxMessage]
  @GuardedBy("this")
  private var client: TransportClient = null
    
//消息发件箱的发送能力
  def send(message: OutboxMessage): Unit = {
    //。。。。其他逻辑
      
    if (dropped) {
      message.onFailure(new SparkException("Message is dropped because Outbox is stopped"))
    } else {
      //清空消息队列：将消息缓存队列中所有消息都发送出去
      drainOutbox()
    }
  }
  
  //清空消息队列
  private def drainOutbox(): Unit = {
    var message: OutboxMessage = null
    synchronized {
      //。。。。其他逻辑
      if (client == null) {
        // 缺少通信客户端，启动连接任务
        launchConnectTask()
        return
      }
      //。。。。其他逻辑
    }
    //。。。。其他逻辑
  }
    
  private def launchConnectTask(): Unit = {
    //由rpcEnv中的客户端连接池负债执行创建通信客户端的线程
    connectFuture = nettyEnv.clientConnectionExecutor.submit(new Callable[Unit] {

      override def call(): Unit = {
        try {
          //根据远端地址建立连接远端的通信客户端
          val _client = nettyEnv.createClient(address)
          //。。。。其他逻辑
        } catch {
          //。。。。其他逻辑
        }
        //。。。。其他逻辑
      }
    })
  }
}

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {
    
  private val clientFactory = transportContext.createClientFactory(createClientBootstraps())
    
  private[netty] def createClient(address: RpcAddress): TransportClient = {
    //通过工厂创建客户端
    clientFactory.createClient(address.host, address.port)
  }
}

入口二：

​ TransportContext#initializePipeline() —> TransportContext#createChannelHandler() —> new TransportClient(channel, responseHandler);

​ 在通过TransportContext初始化通信管道时，需要构建TransportChannelHandler对象添加到socketChannel中，在构建TransportChannelHandler对象时，通过TransportClient构造方法先构建TransportClient对象；

public class TransportContext {
  
  //在TransportContext实例化时，指定rpcHandler为NettyRpcHandler，在NettyRpcHandler上绑定dispatcher、streamManager
  private final RpcHandler rpcHandler;
  
  //初始化管道
  public TransportChannelHandler initializePipeline(
      SocketChannel channel,
      RpcHandler channelRpcHandler) {
    try {
      //创建channelhandler
      TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
        
      channel.pipeline()
        .addLast("encoder", ENCODER)
        .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
        .addLast("decoder", DECODER)
        .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
      	//将channelHandler添加到socketChannel中
        .addLast("handler", channelHandler);
      return channelHandler;
    } catch (RuntimeException e) {
      logger.error("Error while initializing Netty pipeline", e);
      throw e;
    }
  }
  
  private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
    //创建TransportResponseHandler
    TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
    //创建TransportClient
    TransportClient client = new TransportClient(channel, responseHandler);
    //创建TransportRequestHandler
    TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
      rpcHandler, conf.maxChunksBeingTransferred());
    //将TransportResponseHandler、TransportRequestHandler、TransportClient绑定到TransportChannelHandler
    return new TransportChannelHandler(client, responseHandler, requestHandler,
      conf.connectionTimeoutMs(), closeIdleConnections);
  }
}

2.2.2.2.实例化TransportClient逻辑

获取客户端对象步骤：

​ 1、利用ConcurrentHashMap<SocketAddress, TransportClientFactory.ClientPool>缓存通信地址与客户端池映射，客户端池中存放多个客户端，每次从客户端池中随机取出一个客户端对象使用；

​ 2、创建通信客户端对象时，先从ConcurrentHashMap缓存中根据通信地址获取客户端池ClientPool，然后从客户端池ClientPool中随机获取一个客户端对象；

​ 3、如果客户端对象不为空切处于活动状态，更新客户端绑定的TransportChannelHandler对象中TransportResponseHandler对象的最后请求时间；

​ 4、如果客户端对象为空或处于非活动状态，创建一个客户端，将新客户端放入客户端池中，并返回；

新建客户端对象步骤：

​ 1、构建根引导器Bootstrap并对其进行配置；
​ 2、为根引导程序设置管道初始化回调函数，此回调函数将调用TransportContext的initializePipeline方法初始化Channel的pipeline（构建TransportChannelHandler对象添加到socketChannel中，在构建TransportChannelHandler对象时，通过TransportClient构造方法先构建TransportClient对象）；
​ 3、使用根引导程序连接远程服务器，当连接成功对管道初始化时会回调初始化回调函数，将TransportClient和Channel对象分别设置到原子引用clientRef与channelRef中；
​ 4、给TransportClient设置客户端引导程序，即设置TransportClientFactory中的TransportClientBootstrap列表；
​ 5、最后返回此TransportClient对象

总结：

​ 创建TransportClient对象client时，将SocketChannel对象channel绑定到client的属性中；对channel的初始化中，定义了对管道中消息/数据的处理流程，包括编码、解码等，在管道消息处理流程的最后一步是TransportChannelHandler对象；

​ 即通过socketChannel进行消息/数据传递时，最后都会有TransportChannelHandler类进行消息/数据处理；

public class TransportClientFactory implements Closeable {
    
    //获取客户端对象
    public TransportClient createClient(String remoteHost, int remotePort) throws IOException, InterruptedException {
        //封装通信地址
        InetSocketAddress unresolvedAddress = InetSocketAddress.createUnresolved(remoteHost, remotePort);
        //通连接池中获取客户端池
        TransportClientFactory.ClientPool clientPool = (TransportClientFactory.ClientPool)this.connectionPool.get(unresolvedAddress);
        //客户端池不存在，构建客户端池放入连接池中
        if (clientPool == null) {
            this.connectionPool.putIfAbsent(unresolvedAddress, new TransportClientFactory.ClientPool(this.numConnectionsPerPeer));
            clientPool = (TransportClientFactory.ClientPool)this.connectionPool.get(unresolvedAddress);
        }

        //从客户端池中随机获取通信客户端对象client
        int clientIndex = this.rand.nextInt(this.numConnectionsPerPeer);
        TransportClient cachedClient = clientPool.clients[clientIndex];
        if (cachedClient != null && cachedClient.isActive()) {
            //更新客户端绑定的TransportChannelHandler中TransportResponseHandler对象的最后请求时间
            TransportChannelHandler handler = (TransportChannelHandler)cachedClient.getChannel().pipeline().get(TransportChannelHandler.class);
            synchronized(handler) {
                handler.getResponseHandler().updateTimeOfLastRequest();
            }

            //返回通信客户端对象
            if (cachedClient.isActive()) {
                logger.trace("Returning cached connection to {}: {}", cachedClient.getSocketAddress(), cachedClient);
                return cachedClient;
            }
        }

        long preResolveHost = System.nanoTime();
        InetSocketAddress resolvedAddress = new InetSocketAddress(remoteHost, remotePort);
        long hostResolveTimeMs = (System.nanoTime() - preResolveHost) / 1000000L;
        if (hostResolveTimeMs > 2000L) {
            logger.warn("DNS resolution for {} took {} ms", resolvedAddress, hostResolveTimeMs);
        } else {
            logger.trace("DNS resolution for {} took {} ms", resolvedAddress, hostResolveTimeMs);
        }

        synchronized(clientPool.locks[clientIndex]) {
            cachedClient = clientPool.clients[clientIndex];
            if (cachedClient != null) {
                if (cachedClient.isActive()) {
                    logger.trace("Returning cached connection to {}: {}", resolvedAddress, cachedClient);
                    return cachedClient;
                }

                logger.info("Found inactive connection to {}, creating a new one.", resolvedAddress);
            }

            //从客户端池中获取的客户端对象为空或不是活动状态，重新创建一个客户端对象，并放入客户端池
            clientPool.clients[clientIndex] = this.createClient(resolvedAddress);
            return clientPool.clients[clientIndex];
        }
    }
    
    //创建通信客户端对象
    private TransportClient createClient(InetSocketAddress address) throws IOException, InterruptedException {
        logger.debug("Creating new connection to {}", address);
        
        // 构建根引导器Bootstrap并对其进行配置
        Bootstrap bootstrap = new Bootstrap();
    	bootstrap.group(workerGroup)
      		.channel(socketChannelClass)
      		// Disable Nagle's Algorithm since we don't want packets to wait
     		.option(ChannelOption.TCP_NODELAY, true)
      		.option(ChannelOption.SO_KEEPALIVE, true)
      		.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, conf.connectionTimeoutMs())
      		.option(ChannelOption.ALLOCATOR, pooledAllocator);

		if (this.conf.receiveBuf() > 0) {
            bootstrap.option(ChannelOption.SO_RCVBUF, this.conf.receiveBuf());
        }

        if (this.conf.sendBuf() > 0) {
            bootstrap.option(ChannelOption.SO_SNDBUF, this.conf.sendBuf());
        }

        final AtomicReference<TransportClient> clientRef = new AtomicReference();
        final AtomicReference<Channel> channelRef = new AtomicReference();
        
        // 为根引导程序设置管道初始化回调函数
        bootstrap.handler(new ChannelInitializer<SocketChannel>() {
            public void initChannel(SocketChannel ch) {
                //初始化管道：构建TransportChannelHandler对象添加到socketChannel中，在构建TransportChannelHandler对象时，通过TransportClient构造方法先构建TransportClient对象
                TransportChannelHandler clientHandler = TransportClientFactory.this.context.initializePipeline(ch);
                clientRef.set(clientHandler.getClient());
                channelRef.set(ch);
            }
        });
        long preConnect = System.nanoTime();
        
        // 使用根引导程序连接远程服务器
        ChannelFuture cf = bootstrap.connect(address);
        if (!cf.await((long)this.conf.connectionTimeoutMs())) {
            throw new IOException(String.format("Connecting to %s timed out (%s ms)", address, this.conf.connectionTimeoutMs()));
        } else if (cf.cause() != null) {
            throw new IOException(String.format("Failed to connect to %s", address), cf.cause());
        } else {
            TransportClient client = (TransportClient)clientRef.get();
            Channel channel = (Channel)channelRef.get();

            assert client != null : "Channel future completed successfully with null client";

            long preBootstrap = System.nanoTime();
            logger.debug("Connection to {} successful, running bootstraps...", address);

            try {
                Iterator var12 = this.clientBootstraps.iterator();

                while(var12.hasNext()) {
                    // 给TransportClient设置客户端引导程序
                    TransportClientBootstrap clientBootstrap = (TransportClientBootstrap)var12.next();
                    clientBootstrap.doBootstrap(client, channel);
                }
            } catch (Exception var15) {
                long bootstrapTimeMs = (System.nanoTime() - preBootstrap) / 1000000L;
                logger.error("Exception while bootstrapping client after " + bootstrapTimeMs + " ms", var15);
                client.close();
                throw Throwables.propagate(var15);
            }

            long postBootstrap = System.nanoTime();
            logger.info("Successfully created connection to {} after {} ms ({} ms spent in bootstraps)", new Object[]{address, (postBootstrap - preConnect) / 1000000L, (postBootstrap - preBootstrap) / 1000000L});
            return client;
        }
    }
}

2.2.3.TransportClient通过TransportServer发送信息到对应节点

2.2.3.1.TransportClient对象发送消息

通过TransportClient对象绑定的Channel对象，将封装后的消息写入通信管道中；

public class TransportClient implements Closeable {
    public long sendRpc(ByteBuffer message, RpcResponseCallback callback) {
        if (logger.isTraceEnabled()) {
            logger.trace("Sending RPC to {}", NettyUtils.getRemoteAddress(this.channel));
        }

        long requestId = requestId();
        this.handler.addRpcRequest(requestId, callback);
        //监听通信管道
        TransportClient.RpcChannelListener listener = new TransportClient.RpcChannelListener(requestId, callback);
        //封装消息，将消息写入通信管道中，并添加监听
        this.channel.writeAndFlush(new RpcRequest(requestId, new NioManagedBuffer(message))).addListener(listener);
        return requestId;
    }
    
    public void send(ByteBuffer message) {
        //封装消息，将消息写入通信管道中
        this.channel.writeAndFlush(new OneWayMessage(new NioManagedBuffer(message)));
    }
}

2.2.3.2.TransportChannelHandler处理消息

​ 在启动server设置引导程序时，有进行管道初始化工作，其中TransportChannelHandler对象时定义管道处理数据流程的最后一环；即通过管道进行通信时，最后都由TransportChannelHandler对象进行数据处理；

​ 针对rpc请求类消息，由NettyRpcHandler#receive()进行消息处理，最后交给消息分发器进行后续处理

public class TransportChannelHandler extends ChannelInboundHandlerAdapter {
    
    private final TransportClient client;
    private final TransportResponseHandler responseHandler;
    private final TransportRequestHandler requestHandler;
    
    //消息分类，分别由对应消息处理器处理
    public void channelRead(ChannelHandlerContext ctx, Object request) throws Exception {
        if (request instanceof RequestMessage) {
            //请求消息，由请求处理器处理
            this.requestHandler.handle((RequestMessage)request);
        } else if (request instanceof ResponseMessage) {
            //响应消息，由响应处理器处理
            this.responseHandler.handle((ResponseMessage)request);
        } else {
            ctx.fireChannelRead(request);
        }

    }
}

//请求消息处理器
public class TransportRequestHandler extends MessageHandler<RequestMessage> {
    public void handle(RequestMessage request) {
        if (request instanceof ChunkFetchRequest) {
            this.processFetchRequest((ChunkFetchRequest)request);
        } else if (request instanceof RpcRequest) {
            //处理rpc请求
            this.processRpcRequest((RpcRequest)request);
        } else if (request instanceof OneWayMessage) {
            this.processOneWayMessage((OneWayMessage)request);
        } else if (request instanceof StreamRequest) {
            this.processStreamRequest((StreamRequest)request);
        } else {
            if (!(request instanceof UploadStream)) {
                throw new IllegalArgumentException("Unknown request type: " + request);
            }

            this.processStreamUpload((UploadStream)request);
        }

    }
    
    //处理rpc请求
    private void processRpcRequest(final RpcRequest req) {
        try {
            //由NettyRpcHandler#receive()进行消息处理
            this.rpcHandler.receive(this.reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() {
                public void onSuccess(ByteBuffer response) {
                    TransportRequestHandler.this.respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
                }

                public void onFailure(Throwable e) {
                    TransportRequestHandler.this.respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
                }
            });
        } catch (Exception var6) {
            logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, var6);
            this.respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(var6)));
        } finally {
            req.body().release();
        }

    }
}

private[netty] class NettyRpcHandler(
    dispatcher: Dispatcher,
    nettyEnv: NettyRpcEnv,
    streamManager: StreamManager) extends RpcHandler with Logging {

  // A variable to track the remote RpcEnv addresses of all clients
  private val remoteAddresses = new ConcurrentHashMap[RpcAddress, RpcAddress]()

  override def receive(
      client: TransportClient,
      message: ByteBuffer,
      callback: RpcResponseCallback): Unit = {
    val messageToDispatch = internalReceive(client, message)
    //将消息交由消息分发器的postRemoteMessage()方法
    dispatcher.postRemoteMessage(messageToDispatch, callback)
  }
}

//消息分发器处理远端rpc消息
private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {
    def postRemoteMessage(message: RequestMessage, callback: RpcResponseCallback): Unit = {
    val rpcCallContext =
      new RemoteNettyRpcCallContext(nettyEnv, callback, message.senderAddress)
    val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
    postMessage(message.receiver.name, rpcMessage, (e) => callback.onFailure(e))
  }
}

2.2.3.3.总结

rpc消息处理流程：

​ 1、client发送消息到消息管道；

​ 2、消息管道根据定义的消息处理流程进行消息处理，在最后一步通过TransportChannelHandler进行消息分类处理；

​ 3、rpc请求消息在分类后，由TransportRequestHandler进行处理：调用NettyRpcHandler#receive()进行消息处理；

​ 4、在NettyRpcHandler#receive()中，将消息交给消息分发器；由分发器进行后续消息处理；

3.总结

rpc消息通信流程：

针对发送到本地节点的消息，直接交由消息分发器进行消息分发到对应节点的收件箱；

针对发送到远端节点的消息，先放到发件节点的发件箱，然后由通信client推送到通信管道channel；管道中的消息进过编码解码等一系列流程，最后由TransportChannelHandler对象分类处理：针对rpc请求消息，由TransportRequestHandler对象交给NettyRpcHandler处理，NettyRpcHandler将消息交给消息分发器进行消息分发到对应节点的收件箱；

针对各个节点收件箱中消息队列的处理，在消息分发器dispatcher中维护了一个线程池，池中多个线程轮询收件节点，将收件节点收件箱的消息队列的头消息进行分发，调用对应节点进行后续消息处理；

4.参考资料

spark 通信原理源码分析

Spark netty RPC 通信原理

Spark 2.X RPC通信原理

Spark2.0.2源码分析——RPC 通信机制(消息处理)

Spark RPC 通信机制

Spark 通信篇 | spark RPC 模块篇

Spark Network 模块分析

Spark RPC之RpcRequest请求处理流程

Spark2.1.0——内置RPC框架详解