spark源码解析-分析一次完整的远程请求过程

spark版本： 2.0.0

1.概念

1.引入

前面已经介绍了master启动，worker启动和rpc原理，现在结合这些，来探究一下一次完整的远程请求到底是咋样的？就以worker启动后注册到master为例，我们来细细品味一下其远程服务调用过程。

2.远程服务请求过程

worker注册到master的方法是masterEndpoint.ask[RegisterWorkerResponse](RegisterWorker( workerId, host, port, self, cores, memory, workerWebUiUrl)，即worker通过访问master服务然后根据master的结果，worker也响应的处理。
在rpc原理博客中，可以知道这代码最终是将message=RegisterWorker添加到outbox中，然后取出来通过RpcOutboxMessage.sendWith方法发送到服务端（master endpoint），在sendWith方法中调用了sendRpc方法中的channel.writeAndFlush(new RpcRequest(requestId, new NioManagedBuffer(message)))将数据以RpcRequest请求类型发送到master endpoint。接下来会经过一个关键的handler:MessageEncoder将请求对象转换为字节数组，然后master endpoint收到字节数组之后，先通过MessageDecoder的decode方法转为RpcRequest对象。(TransportFrameDecoder是处理粘包拆包不是重点)

  private Message decode(Message.Type msgType, ByteBuf in) {
    switch (msgType) {

      case RpcRequest:
        return RpcRequest.decode(in);

然后依次通过IdleStateHandler，TransportChannelHandler这两个handler，最主要的是TransportChannelHandler，它将告诉我们最终的调用过程。

TransportChannelHandler.java
-----------------------

  public void channelRead0(ChannelHandlerContext ctx, Message request) throws Exception {
    // 区分消息类型
    if (request instanceof RequestMessage) {
      requestHandler.handle((RequestMessage) request);

TransportRequestHandler.java
----------------------

  public void handle(RequestMessage request) {
    ...
    else if (request instanceof RpcRequest) {
      processRpcRequest((RpcRequest) request);
 
 
    
  /**
   * 处理rpc请求
   * @param req
   */
  private void processRpcRequest(final RpcRequest req) {
    try {
      // rpcHandler=NettyRpcHandler
      rpcHandler.receive(reverseClient, req.body().nioByteBuffer(), new RpcResponseCallback() {
        @Override
        public void onSuccess(ByteBuffer response) {
          respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
        }

        @Override
        public void onFailure(Throwable e) {
          respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
        }
      });
    } catch (Exception e) {
      logger.error("Error while invoking RpcHandler#receive() on RPC id " + req.requestId, e);
      respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
    } finally {
      req.body().release();
    }
  }

可以看出请求处理逻辑是通过：rpcHandler.receive方法（rpcHandler=NettyRpcHandler）

NettyRpcEnv.scala
-------------------

  override def receive(
      client: TransportClient,
      message: ByteBuffer,
      callback: RpcResponseCallback): Unit = {
    // ByteBuffer => requestMessage
    val messageToDispatch = internalReceive(client, message)
    // 分发请求信息
    dispatcher.postRemoteMessage(messageToDispatch, callback)
  }
  
  
  Dispatcher.scala
  -------------------
  
  
  def postRemoteMessage(message: RequestMessage, callback: RpcResponseCallback): Unit = {
      val rpcCallContext =
      new RemoteNettyRpcCallContext(nettyEnv, callback, message.senderAddress)
    val rpcMessage = RpcMessage(message.senderAddress, message.content, rpcCallContext)
    postMessage(message.receiver.name, rpcMessage, (e) => callback.onFailure(e))
  }

postMessage这个方法前面已经介绍了就是消息会发送到inbox中，然后自动调用masterEndpoint.receiveAndReply（因为请求类型是RpcMessage），但是异常情况没有说明，现在来看一下：

  private def postMessage(
      endpointName: String,
      message: InboxMessage,
      callbackIfStopped: (Exception) => Unit): Unit = {
    val error = synchronized {
      // 获取endpoint的信息
      val data = endpoints.get(endpointName)
      if (stopped) {
        Some(new RpcEnvStoppedException())
      } else if (data == null) {
        Some(new SparkException(s"Could not find $endpointName."))
      } else {
        // 往需要发送的通信端inbox中添加一条消息，并添加到receivers从而触发消息处理
        data.inbox.post(message)
        receivers.offer(data)
        None
      }
    }
    // 错误处理核心
    error.foreach(callbackIfStopped)
  }

注意错误处理代码：callbackIfStopped是

TransportRequestHandler.java
----------------------

new RpcResponseCallback() {
        @Override
        public void onSuccess(ByteBuffer response) {
          respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
        }

        @Override
        public void onFailure(Throwable e) {
          respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
        }
      }

最终会调用TransportRequestHandler.respond方法

  private void respond(final Encodable result) {
    final String remoteAddress = channel.remoteAddress().toString();
    channel.writeAndFlush(result).addListener(
      new ChannelFutureListener() {
        @Override
        public void operationComplete(ChannelFuture future) throws Exception {
          if (future.isSuccess()) {
            logger.trace(String.format("Sent result %s to client %s", result, remoteAddress));
          } else {
            logger.error(String.format("Error sending result %s to %s; closing connection",
              result, remoteAddress), future.cause());
            channel.close();
          }
        }
      }
    );
  }

即将错误信息返回到请求方。
如果请求是正常的最终会调用Master.receiveAndReply请求类型是RegisterWorker：

Master.scala
--------------------

 /**
    * 请求并返回
    * @param context
    * @return
    */
  override def receiveAndReply(context: RpcCallContext): PartialFunction[Any, Unit] = {
    case RegisterWorker(
        id, workerHost, workerPort, workerRef, cores, memory, workerWebUiUrl) =>
      logInfo("Registering worker %s:%d with %d cores, %s RAM".format(
        workerHost, workerPort, cores, Utils.megabytesToString(memory)))
      // 如果master是STANDBY，直接返回MasterInStandby消息
      if (state == RecoveryState.STANDBY) {
        context.reply(MasterInStandby)
        // 如果已经存在相同的workerid
      } else if (idToWorker.contains(id)) {
        context.reply(RegisterWorkerFailed("Duplicate worker ID"))
      } else {
        // 封装worker信息对象
        val worker = new WorkerInfo(id, workerHost, workerPort, cores, memory,
          workerRef, workerWebUiUrl)
        // 注册worker并修改对应的属性
        if (registerWorker(worker)) {
          persistenceEngine.addWorker(worker)
          // 响应master的web ui地址
          context.reply(RegisteredWorker(self, masterWebUiUrl))
          
          // 应用程序的资源调度（比较复杂，后面介绍）
          schedule()
        } else {
          val workerAddress = worker.endpoint.address
          logWarning("Worker registration failed. Attempted to re-register worker at same " +
            "address: " + workerAddress)
          context.reply(RegisterWorkerFailed("Attempted to re-register worker at same address: "
            + workerAddress))
        }
      }

上面的处理逻辑比较简单，就是master根据worker的信息，做不同的处理。但是最终都会调用方法context.reply,首先我们需要确认context对象是什么，在前面方法postRemoteMessage中创建了该对象RemoteNettyRpcCallContext，所以context即RemoteNettyRpcCallContext。根据代码调用可知，最终会调用以下代码:

NettyRpcCallContext.scala
-----------------------


  override protected def send(message: Any): Unit = {
    // 序列化message
    val reply = nettyEnv.serialize(message)
    // 调用callback的onSuccess方法发送信息
    callback.onSuccess(reply)
  }

callback在前面已经分析过了，就是TransportRequestHandler中的匿名内部类RpcResponseCallback的对象。

TransportRequestHandler.java
----------------------

new RpcResponseCallback() {
        @Override
        public void onSuccess(ByteBuffer response) {
          respond(new RpcResponse(req.requestId, new NioManagedBuffer(response)));
        }

        @Override
        public void onFailure(Throwable e) {
          respond(new RpcFailure(req.requestId, Throwables.getStackTraceAsString(e)));
        }
      }

所以master就会将处理后的结果返回到worker服务（注意：Worker的注册请求产生的请求对象是RpcRequest对象，master响应使用的是RpcResponse/RpcFailure对象）

3.总结

通过上面对worker注册到master endpoint的远程请求过程介绍，可以使用下面的示意图表示：