Spark集群创建之核心角色的创建过程

基本概念

这里所说的核心角色，是指如Master、Worker、Client等，这类在各自的进程中需要初始化一个新的RpcEnv环境的角色，他们同时负担了同进程内其它RpcEndpoint与远程端点的RPC消息交互。所有这些核心角色的创建流程大体相同，只是具体处理消息的方法不同，因此这里以Client为例，浅析其创建时细节。

RpcEnv

不论Driver进程、Master进程、Worker进程等，但凡是在同一个Spark进程环境中生成的、需要消息交互的对象（RpcEndpoint），都共用一个RpcEnv实例，以便使用统一的环境信息来收发RPC消息，同时这些RpcEndpoint必须显示调用setupEndpoint(...)方法完成注册。
RpcEnv类的核心定义描述及相关代码如下：

传递从远程RpcEndpointRef收到的消息到指定的已经注册在当前环境的RpcEndpoint（在下一小节分析）
查找已经在当前环境中注册的RpcEndpoint
序列化消息体
如果当前RpcEnv是一个Server端点，返回文件服务对象，以提供文件下载服务
作为client，创建一个下载文件的通道

private[spark] abstract class RpcEnv(conf: SparkConf) {

  private[spark] val defaultLookupTimeout = RpcUtils.lookupRpcTimeout(conf)

  private[rpc] def endpointRef(endpoint: RpcEndpoint): RpcEndpointRef

  def address: RpcAddress

  def setupEndpoint(name: String, endpoint: RpcEndpoint): RpcEndpointRef

  def asyncSetupEndpointRefByURI(uri: String): Future[RpcEndpointRef]

  def setupEndpointRefByURI(uri: String): RpcEndpointRef = {
    defaultLookupTimeout.awaitResult(asyncSetupEndpointRefByURI(uri))
  }

  def setupEndpointRef(address: RpcAddress, endpointName: String): RpcEndpointRef = {
    setupEndpointRefByURI(RpcEndpointAddress(address, endpointName).toString)
  }

  def stop(endpoint: RpcEndpointRef): Unit

  def shutdown(): Unit

  def awaitTermination(): Unit

  def deserialize[T](deserializationAction: () => T): T

  def fileServer: RpcEnvFileServer

  def openChannel(uri: String): ReadableByteChannel
}

NettyRpcEnv

RpcEnv的一个实现类，底层通过Netty库来完成消息的交互，该类封装了Rpc消息交换所需的各种组件，并提供了接收、发送消息的方法，核心的类定义摘取如下：

private[netty] class NettyRpcEnv(
    val conf: SparkConf,
    javaSerializerInstance: JavaSerializerInstance,
    host: String,
    securityManager: SecurityManager,
    numUsableCores: Int) extends RpcEnv(conf) with Logging {

  // 生成RPC相关的一些配置信息
  private[netty] val transportConf = SparkTransportConf.fromSparkConf(
    conf.clone.set("spark.rpc.io.numConnectionsPerPeer", "1"),
    "rpc",
    conf.getInt("spark.rpc.io.threads", numUsableCores))
  // 消息转发器，记录了所有在当前环境中注册的RpcEndpoint及对应的引用RpcEndpointRef
  // 为了便于把所有的消息与对应的RpcEndpoint的绑定，内部会抽象一个RpcEndpointData的
  // 类，用于保存RpcEndpoint及其所有的消息，详细的分析见后面小节
  private val dispatcher: Dispatcher = new Dispatcher(this, numUsableCores)
  // Netty实现的流数据文件管理器，主要用于管理文件数据的传输
  private val streamManager = new NettyStreamManager(this)
  
  // 消息传输上下文管理器，用来创建底层的网络通讯服务器对象、客户端对象及相应的消息处理
  // 器（TransportChannelHandler），基于Netty
  private val transportContext = new TransportContext(transportConf,
    new NettyRpcHandler(dispatcher, this, streamManager))
  // 生成客户端工厂对象，该工厂对象用于生成Netty客户端，并连接到指定的服务器地址
  private val clientFactory = transportContext.createClientFactory(createClientBootstraps())

  // 一个独立的Netty客户端工厂对象，用来支持文件下载，这可以避免与处理RPC消息主流程，使
  // 用同一个处理器，并根据需要配置不同的参数。
  @volatile private var fileDownloadFactory: TransportClientFactory = _

  // 定时调度线程，当处理需要ACK的消息时，可能会出现消息处理超时的情况，为了能够在发生超
  // 时的时候做一些额外的工作，可以通过调用该线程池执行器来启动一个定时任务检测当前的RPC
  // 消息是否超时
  val timeoutScheduler = ThreadUtils.newDaemonSingleThreadScheduledExecutor("netty-rpc-env-timeout")
  // 为了能够接收来自其它RpcEndpoint的请求，需要内部创建一个Netty服务器对象，
  // 用来监听端口及获取消息
  @volatile private var server: TransportServer = _

  private val stopped = new AtomicBoolean(false)
  // 
  // 当当前RPC端点想要发送消息到时远程结点时，比如调用send(..)/ask(...)方法，会根据
  // 消息体中的目标主机地址，在此Map中查找对应的Outbox对象（Outbox可以看成一个消息队
  // 列），最终将消息添加到此队列里，以便无阻塞地发送消息。
  private val outboxes = new ConcurrentHashMap[RpcAddress, Outbox]()

  // 许多具体的成员函数，暂忽略
  // ...
}

Dispatcher

消息转发器，用用于接收发送到时所有注册到当前Dispatcher对象中的RpcEndpoint的消息，然后根据接收消息中携带的Endpoint Name，将消息转发到对应的RpcEndpoint对象进行处理。
Dispather主要做三件事：

• RpcEndpoint 注册/反注册，由于一个Endpoint可能会接收多条消息，因此需要有一个队列来保存所有发送给当前Endpoint的消息，因此在注册时会将每一个RpcEndpoint封装成一个EndpointData对象，保存到Map数据结构中。
• 提供一些门面函数post*(...)，用于当NettyRpcEnv创建的底层Netty消息监听线程接收到的服务端的消息时，通过这些函数来转发到对应的RpcEndpoint对应的消息队列里
• 提供一个消息轮循函数，由于当接收到新的RPC消息时，其内部维护的线性阻塞队列receivers被添加一个新的EndpointData的引用，此函数便用于从队列中取出这些引用并处理。

private[netty] class Dispatcher(nettyEnv: NettyRpcEnv, numUsableCores: Int) extends Logging {

  private class EndpointData(
      val name: String,
      val endpoint: RpcEndpoint,
      val ref: NettyRpcEndpointRef) {
    val inbox = new Inbox(ref, endpoint)
  }

  private val endpoints: ConcurrentMap[String, EndpointData] =
    new ConcurrentHashMap[String, EndpointData]
  private val endpointRefs: ConcurrentMap[RpcEndpoint, RpcEndpointRef] =
    new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]

  // Track the receivers whose inboxes may contain messages.
  private val receivers = new LinkedBlockingQueue[EndpointData]

  def registerRpcEndpoint(name: String, endpoint: RpcEndpoint): NettyRpcEndpointRef = {
    val addr = RpcEndpointAddress(nettyEnv.address, name)
    val endpointRef = new NettyRpcEndpointRef(nettyEnv.conf, addr, nettyEnv)
    synchronized {
      if (stopped) {
        throw new IllegalStateException("RpcEnv has been stopped")
      }
      if (endpoints.putIfAbsent(name, new EndpointData(name, endpoint, endpointRef)) != null) {
        throw new IllegalArgumentException(s"There is already an RpcEndpoint called $name")
      }
      val data = endpoints.get(name)
      endpointRefs.put(data.endpoint, data.ref)
      receivers.offer(data)  // for the OnStart message
    }
    endpointRef
  }

  /**
   * Posts a message to a specific endpoint.
   *
   * @param endpointName name of the endpoint.
   * @param message the message to post
   * @param callbackIfStopped callback function if the endpoint is stopped.
   */
  private def postMessage(
      endpointName: String,
      message: InboxMessage,
      callbackIfStopped: (Exception) => Unit): Unit = {
    val error = synchronized {
      val data = endpoints.get(endpointName)
      if (stopped) {
        Some(new RpcEnvStoppedException())
      } else if (data == null) {
        Some(new SparkException(s"Could not find $endpointName."))
      } else {
        data.inbox.post(message)
        receivers.offer(data)
        None
      }
    }
    // We don't need to call `onStop` in the `synchronized` block
    error.foreach(callbackIfStopped)
  }

  /** Thread pool used for dispatching messages. */
  private val threadpool: ThreadPoolExecutor = {
    val availableCores =
      if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
    val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
      math.max(2, availableCores))
    val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
    for (i <- 0 until numThreads) {
      pool.execute(new MessageLoop)
    }
    pool
  }

  /** Message loop used for dispatching messages. */
  private class MessageLoop extends Runnable {
    override def run(): Unit = {
      try {
        while (true) {
          try {
            val data = receivers.take()
            if (data == PoisonPill) {
              // Put PoisonPill back so that other MessageLoops can see it.
              receivers.offer(PoisonPill)
              return
            }
            data.inbox.process(Dispatcher.this)
          } catch {
            case NonFatal(e) => logError(e.getMessage, e)
          }
        }
      } catch {
        case _: InterruptedException => // exit
        case t: Throwable =>
          try {
            // Re-submit a MessageLoop so that Dispatcher will still work if
            // UncaughtExceptionHandler decides to not kill JVM.
            threadpool.execute(new MessageLoop)
          } finally {
            throw t
          }
      }
    }
  }

  /** A poison endpoint that indicates MessageLoop should exit its message loop. */
  private val PoisonPill = new EndpointData(null, null, null)
}

TransportContext

构建底层网络通讯基础组件的上下文管理器，该类是一个创建消息服务器TransportServer、客户端工厂TransportClientFactory的门面类，同时提供了创建统一的消息处理器TransportChannelHandler的方法。
由于不管是Netty服务端的事件池，还是客户端的事件池，都需要用户指定一个Handler来根据不同的消息，执行不同的动作，Spark内部会使用一个统一的包装类TransportChannelHandler，用来监听channel上的消息。当有新消息到来时，根据消息类型的不同，分别转发给TransportResponseHandler或TransportRequestHandler执行具体的操作。

Outbox

当一个RpcEndpoint想要发送消息到远端地址时，会创建一个此类的实例，用于封装目标RPC地址及对应的客户端TransportClient对象，以便在后续单向或双向通讯过程中，通过同一个客户端对象来交换数据，并且在连接中断后，清除该实例。类的核心定义摘取如下：

private[netty] class Outbox(nettyEnv: NettyRpcEnv, val address: RpcAddress) {

  outbox => // Give this an alias so we can use it more clearly in closures.

  @GuardedBy("this")
  private val messages = new java.util.LinkedList[OutboxMessage]

  @GuardedBy("this")
  private var client: TransportClient = null

  /**
   * connectFuture points to the connect task. If there is no connect task, connectFuture will be
   * null.
   */
  @GuardedBy("this")
  private var connectFuture: java.util.concurrent.Future[Unit] = null

  @GuardedBy("this")
  private var stopped = false

  /**
   * If there is any thread draining the message queue
   */
  @GuardedBy("this")
  private var draining = false

  /**
   * Send a message. If there is no active connection, cache it and launch a new connection. If
   * [[Outbox]] is stopped, the sender will be notified with a [[SparkException]].
   */
  def send(message: OutboxMessage): Unit = {...}

  /**
   * Drain the message queue. If there is other draining thread, just exit. If the connection has
   * not been established, launch a task in the `nettyEnv.clientConnectionExecutor` to setup the
   * connection.
   */
  private def drainOutbox(): Unit = {...}

  private def launchConnectTask(): Unit = {...}

  /**
   * Stop [[Inbox]] and notify the waiting messages with the cause.
   */
  private def handleNetworkFailure(e: Throwable): Unit = {...}

  private def closeClient(): Unit = {...}

  /**
   * Stop [[Outbox]]. The remaining messages in the [[Outbox]] will be notified with a
   * [[SparkException]].
   */
  def stop(): Unit = {...}
  }
}

RpcEndpoint

RpcEndpoint类，定义了一组方法，用来处理RPC消息，它保证了constructor -> onStart -> recevie* -> onStop这些方法的有序性，但可以并发地调用receive方法，如果想要线程安全地调用receive方法，则需要继承ThreadSafeRpcEndpoint特性。
此类用来定义一些方法，用来处理收到的消息，它主要包含了以下两个成员变量：

RpcEnv 当前RpcEndpoint实例类所绑定的RPC环境
RpcEndpointRef 当前RpcEndpoint实例的一个引用，此类的实例可以被序列化且是线程安全的。远程客户端可以通过获取此对象，来发送消息。

// 在消息处理时，如果发现当前接收消息的ThreadSafeRpcEndpoint的，则顺序处理消息，
// 否则并发处理
private[spark] trait ThreadSafeRpcEndpoint extends RpcEndpoint
private[spark] trait RpcEndpoint {
  /**
   * The [[RpcEnv]] that this [[RpcEndpoint]] is registered to.
   */
  val rpcEnv: RpcEnv

  /**
   * The [[RpcEndpointRef]] of this [[RpcEndpoint]]. `self` will become valid when `onStart` is
   * called. And `self` will become `null` when `onStop` is called.
   *
   * Note: Because before `onStart`, [[RpcEndpoint]] has not yet been registered and there is not
   * valid [[RpcEndpointRef]] for it. So don't call `self` before `onStart` is called.
   */
  final def self: RpcEndpointRef = {
    require(rpcEnv != null, "rpcEnv has not been initialized")
    rpcEnv.endpointRef(this)
  }

  /**
   * Process messages from `RpcEndpointRef.send` or `RpcCallContext.reply`. If receiving a
   * unmatched message, `SparkException` will be thrown and sent to `onError`.
   */
  def receive: PartialFunction[Any, Unit] = {
    case _ => throw new SparkException(self + " does not implement 'receive'")
  }

  /**
   * Process messages from `RpcEndpointRef.ask`. If receiving a unmatched message,
   * `SparkException` will be thrown and sent to `onError`.
   */
  def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
    case _ => context.sendFailure(new SparkException(self + " won't reply anything"))
  }

  /**
   * Invoked when any exception is thrown during handling messages.
   */
  def onError(cause: Throwable): Unit = {
    // By default, throw e and let RpcEnv handle it
    throw cause
  }

  /**
   * Invoked when `remoteAddress` is connected to the current node.
   */
  def onConnected(remoteAddress: RpcAddress): Unit = {
    // By default, do nothing.
  }

  /**
   * Invoked when `remoteAddress` is lost.
   */
  def onDisconnected(remoteAddress: RpcAddress): Unit = {
    // By default, do nothing.
  }

  /**
   * Invoked when some network error happens in the connection between the current node and
   * `remoteAddress`.
   */
  def onNetworkError(cause: Throwable, remoteAddress: RpcAddress): Unit = {
    // By default, do nothing.
  }

  /**
   * Invoked before [[RpcEndpoint]] starts to handle any message.
   */
  def onStart(): Unit = {
    // By default, do nothing.
  }

  /**
   * Invoked when [[RpcEndpoint]] is stopping. `self` will be `null` in this method and you cannot
   * use it to send or ask messages.
   */
  def onStop(): Unit = {
    // By default, do nothing.
  }

  /**
   * A convenient method to stop [[RpcEndpoint]].
   */
  final def stop(): Unit = {
    val _self = self
    if (_self != null) {
      rpcEnv.stop(_self)
    }
  }
}

UML图

SparkApplication角色关联关系图

创建其它Spark角色时，如Worker、Master，涉及到的内部类关联关系图与此类似，都是在其入口方法内，先创建一个RpcEnv的实例类对象NettryRpcEnv，然后将自己注册到当前环境中，并在内部创建用于接收和转发消息的工具类Dispatcher和发送消息到时远端的工具类Outbox。