Hadoop源码分析（18）

Hadoop源码分析（18）

1、 RPC分析

 在文档（17）中分析了在RPC server在启动的时候会启动四类线程：listener，reader，handler，responder。并详细解析listener的运行过程。

  listener负责监听端口，当接收到请求后listener会将请求封装成connection，并添加到reader的队列中。这里接着分析reader的运行过程。其启动后的run方法如下：

     @Override
      public void run() {
        LOG.info("Starting " + Thread.currentThread().getName());
        try {
          doRunLoop();
        } finally {
          try {
            readSelector.close();
          } catch (IOException ioe) {
            LOG.error("Error closing read selector in " + Thread.currentThread().getName(), ioe);
          }
        }
      }

  这里很简单就是第5行的doRunLoop方法，其内容如下：

 private synchronized void doRunLoop() {
        while (running) {
          SelectionKey key = null;
          try {
            // consume as many connections as currently queued to avoid
            // unbridled acceptance of connections that starves the select
            int size = pendingConnections.size();
            for (int i=size; i>0; i--) {
              Connection conn = pendingConnections.take();
              conn.channel.register(readSelector, SelectionKey.OP_READ, conn);
            }
            readSelector.select();

            Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
            while (iter.hasNext()) {
              key = iter.next();
              iter.remove();
              try {
                if (key.isReadable()) {
                  doRead(key);
                }
              } catch (CancelledKeyException cke) {
                // something else closed the connection, ex. responder or
                // the listener doing an idle scan.  ignore it and let them
                // clean up.
                LOG.info(Thread.currentThread().getName() +
                    ": connection aborted from " + key.attachment());
              }
              key = null;
            }
          } catch (InterruptedException e) {
            if (running) {                      // unexpected -- log it
              LOG.info(Thread.currentThread().getName() + " unexpectedly interrupted", e);
            }
          } catch (IOException ex) {
            LOG.error("Error in Reader", ex);
          } catch (Throwable re) {
            LOG.fatal("Bug in read selector!", re);
            ExitUtil.terminate(1, "Bug in read selector!");
          }
        }
      }

  这个方法也是一个死循环方法，第2行的while循环与listener中的相同。然后是第7行到第11行，首先是第7行获取pendingConnections队列的大小，这个队列在文档（17）中分析过listener会将接收到的请求封装成connection然后添加到这个队列中。当队列的大小不为0的时候，会继续向下执行for循环中的内容。首先是第9行从队列中取出connection，然后是第10行从connection取出channel，将channel注册到readSelector中，注册的事件为read事件。

  然后是第12行到末尾这里与listener中的一样用于处理发生在channel上的事件。因为这里只注册了读事件所以实际只会处理读事件（第19、20行），在处理读事件的时候实际调用的是doRead方法。这个方法内容如下：

void doRead(SelectionKey key) throws InterruptedException {
      int count = 0;
      Connection c = (Connection)key.attachment();
      if (c == null) {
        return;  
      }
      c.setLastContact(Time.now());

      try {
        count = c.readAndProcess();
      } catch (InterruptedException ieo) {
        LOG.info(Thread.currentThread().getName() + ": readAndProcess caught InterruptedException", ieo);
        throw ieo;
      } catch (Exception e) {
        // a WrappedRpcServerException is an exception that has been sent
        // to the client, so the stacktrace is unnecessary; any other
        // exceptions are unexpected internal server errors and thus the
        // stacktrace should be logged
        LOG.info(Thread.currentThread().getName() + ": readAndProcess from client " +
            c.getHostAddress() + " threw exception [" + e + "]",
            (e instanceof WrappedRpcServerException) ? null : e);
        count = -1; //so that the (count < 0) block is executed
      }
      if (count < 0) {
        closeConnection(c);
        c = null;
      }
      else {
        c.setLastContact(Time.now());
      }
    } 

  这个方法的重点就一句，第10行的readAndProcess方法。这个方法是connection的方法，这个connection是从SelectionKey中获取的。 readAndProcess方法内容如下：

 public int readAndProcess()
        throws WrappedRpcServerException, IOException, InterruptedException {
      while (true) {
        /* Read at most one RPC. If the header is not read completely yet
         * then iterate until we read first RPC or until there is no data left.
         */    
        int count = -1;
        if (dataLengthBuffer.remaining() > 0) {
          count = channelRead(channel, dataLengthBuffer);       
          if (count < 0 || dataLengthBuffer.remaining() > 0) 
            return count;
        }

        if (!connectionHeaderRead) {
          //Every connection is expected to send the header.
          if (connectionHeaderBuf == null) {
            connectionHeaderBuf = ByteBuffer.allocate(3);
          }
          count = channelRead(channel, connectionHeaderBuf);
          if (count < 0 || connectionHeaderBuf.remaining() > 0) {
            return count;
          }
          int version = connectionHeaderBuf.get(0);
          // TODO we should add handler for service class later
          this.setServiceClass(connectionHeaderBuf.get(1));
          dataLengthBuffer.flip();

          // Check if it looks like the user is hitting an IPC port
          // with an HTTP GET - this is a common error, so we can
          // send back a simple string indicating as much.
          if (HTTP_GET_BYTES.equals(dataLengthBuffer)) {
            setupHttpRequestOnIpcPortResponse();
            return -1;
          }

          if (!RpcConstants.HEADER.equals(dataLengthBuffer)
              || version != CURRENT_VERSION) {
            //Warning is ok since this is not supposed to happen.
            LOG.warn("Incorrect header or version mismatch from " + 
                     hostAddress + ":" + remotePort +
                     " got version " + version + 
                     " expected version " + CURRENT_VERSION);
            setupBadVersionResponse(version);
            return -1;
          }

          // this may switch us into SIMPLE
          authProtocol = initializeAuthContext(connectionHeaderBuf.get(2));          

          dataLengthBuffer.clear();
          connectionHeaderBuf = null;
          connectionHeaderRead = true;
          continue;
        }

        if (data == null) {
          dataLengthBuffer.flip();
          dataLength = dataLengthBuffer.getInt();
          checkDataLength(dataLength);
          data = ByteBuffer.allocate(dataLength);
        }

        count = channelRead(channel, data);

        if (data.remaining() == 0) {
          dataLengthBuffer.clear();
          data.flip();
          boolean isHeaderRead = connectionContextRead;
          processOneRpc(data.array());
          data = null;
          if (!isHeaderRead) {
            continue;
          }
        } 
        return count;
      }
    }

 这段代码中执行读取任务的代码被放在了一个死循环中，若代码执行成功则会执行第75行的return退出循环。

  在文档（15）中解析了发送请求的sendRpcRequest方法，在这个方法里发送的数据内容如下：

  从上图中可以看出这里首先是写入了一个totalLength，然后才是数据data，而在data里又分成两部分：header和request。data中内容如下：

  所以从channel中发送的数据可以分成三部分。首先是表示整个数据长度的totalLength，然后是表示请求头的header，最后是具体的请求request。

  所以在readAndProcess方法中读取数据也分为三部分。首先是第7行到第12行，这里会从channel中读取totalLength。读取数据使用的是channelRead方法，这里方法会从指定的channel中读取数据到指定的buffer中。然后是第14行到第54行，这段代码和上述解析的没有太大关系，它是用来处理connection刚连接时候发送的测试请求的，这段代码只会在连接刚建立的时候执行一次。然后是56行到63行，这里会更totalLength来创建一个buffer，然后利用channelRead方法将数据都读取到buffer中。最后是第65行到末尾，调用processOneRpc方法解析buffer中的数据。

  processOneRpc方法内容如下：

private void processOneRpc(byte[] buf)
        throws IOException, WrappedRpcServerException, InterruptedException {
      int callId = -1;
      int retry = RpcConstants.INVALID_RETRY_COUNT;
      try {
        final DataInputStream dis =
            new DataInputStream(new ByteArrayInputStream(buf));
        final RpcRequestHeaderProto header =
            decodeProtobufFromStream(RpcRequestHeaderProto.newBuilder(), dis);
        callId = header.getCallId();
        retry = header.getRetryCount();
        if (LOG.isDebugEnabled()) {
          LOG.debug(" got #" + callId);
        }
        checkRpcHeaders(header);

        if (callId < 0) { // callIds typically used during connection setup
          processRpcOutOfBandRequest(header, dis);
        } else if (!connectionContextRead) {
          throw new WrappedRpcServerException(
              RpcErrorCodeProto.FATAL_INVALID_RPC_HEADER,
              "Connection context not established");
        } else {
          processRpcRequest(header, dis);
        }
      } catch (WrappedRpcServerException wrse) { // inform client of error
        Throwable ioe = wrse.getCause();
        final Call call = new Call(callId, retry, null, this);
        setupResponse(authFailedResponse, call,
            RpcStatusProto.FATAL, wrse.getRpcErrorCodeProto(), null,
            ioe.getClass().getName(), ioe.getMessage());
        call.sendResponse();
        throw wrse;
      }
    }

  在上文解析了buffer中的内容分为两部分：header和request。在第6行会先将buffer封装成输入流，然后是第8行从输入流中解析出header的内容。然后是第10行，从header中拿到callId。最后是第17行到第25行，根据callid的值来判断该执行那个方法，正常情况下是执行第24行的processRpcRequest方法。这个方法内容如下：

 private void processRpcRequest(RpcRequestHeaderProto header,
        DataInputStream dis) throws WrappedRpcServerException,
        InterruptedException {
      Class<? extends Writable> rpcRequestClass = 
          getRpcRequestWrapper(header.getRpcKind());
      if (rpcRequestClass == null) {
        LOG.warn("Unknown rpc kind "  + header.getRpcKind() + 
            " from client " + getHostAddress());
        final String err = "Unknown rpc kind in rpc header"  + 
            header.getRpcKind();
        throw new WrappedRpcServerException(
            RpcErrorCodeProto.FATAL_INVALID_RPC_HEADER, err);   
      }
      Writable rpcRequest;
      try { //Read the rpc request
        rpcRequest = ReflectionUtils.newInstance(rpcRequestClass, conf);
        rpcRequest.readFields(dis);
      } catch (Throwable t) { // includes runtime exception from newInstance
        LOG.warn("Unable to read call parameters for client " +
                 getHostAddress() + "on connection protocol " +
            this.protocolName + " for rpcKind " + header.getRpcKind(),  t);
        String err = "IPC server unable to read call parameters: "+ t.getMessage();
        throw new WrappedRpcServerException(
            RpcErrorCodeProto.FATAL_DESERIALIZING_REQUEST, err);
      }

      Span traceSpan = null;
      if (header.hasTraceInfo()) {
        // If the incoming RPC included tracing info, always continue the trace
        TraceInfo parentSpan = new TraceInfo(header.getTraceInfo().getTraceId(),
                                             header.getTraceInfo().getParentId());
        traceSpan = Trace.startSpan(rpcRequest.toString(), parentSpan).detach();
      }

      Call call = new Call(header.getCallId(), header.getRetryCount(),
          rpcRequest, this, ProtoUtil.convert(header.getRpcKind()),
          header.getClientId().toByteArray(), traceSpan);

      if (callQueue.isClientBackoffEnabled()) {
        // if RPC queue is full, we will ask the RPC client to back off by
        // throwing RetriableException. Whether RPC client will honor
        // RetriableException and retry depends on client ipc retry policy.
        // For example, FailoverOnNetworkExceptionRetry handles
        // RetriableException.
        queueRequestOrAskClientToBackOff(call);
      } else {
        callQueue.put(call);              // queue the call; maybe blocked here
      }
      incRpcCount();  // Increment the rpc count
    }

  首先是第14行到第17行，这里会根据header里的信息继续解析buffer中剩下的request。然后是第35行将解析好的参数封装成一个call对象。最后是第47行将创建的call对象放入callQueue中。