聊聊Thrift(四) thrift 服务篇-TNonblockingServer

聊聊Thrift(四) thrift 服务篇-TNonblockingServer

TNonblockingServer是thrift提供的NIO实现的服务模式，提供非阻塞的服务处理，用一个单线程来处理所有的RPC请求。
类关系如下：

AbstractNonblockingServer是TNonblockingServer和TThreadedSelectorServer的父类，TThreadedSelectorServer即为我们下篇将要讲解的高级非阻塞模式
class结构如下图：

Args是静态内部类，用于传递参数，通过Args可以设置读缓冲区大小，序列化方式，传输协议以及RPC处理类。

通过AbstractNonblockingServer 的serve方法，启动服务，启动接受线程，启动监听，等待服务结束

 /**
   * Begin accepting connections and processing invocations.
   */
  public void serve() {
    // start any IO threads
    if (!startThreads()) {
      return;
    }

    // start listening, or exit
    if (!startListening()) {
      return;
    }

    setServing(true);

    // this will block while we serve
    waitForShutdown();

    setServing(false);

    // do a little cleanup
    stopListening();
  }

重点需要我们关注的是startThreads，在startThreads的实现中，会创建一个SelectAcceptThread线程，并启动。

protected boolean startThreads() {
  // start the selector  
  try {
    selectAcceptThread_ = new SelectAcceptThread((TNonblockingServerTransport)serverTransport_);    
    return true;  
} catch (IOException e) {
    LOGGER.error("Failed to start selector thread!", e);    
    return false;  }
}

SelectAcceptThread就是我们的NIO处理线程，用来accept客户端连接，读写请求

流程：startThreads-》SelectAcceptThread-》select
其中select用来处理所有io事件，select通过selector.select()等待IO事件，然后通过handleAccept、handleRead、handleWrite处理接收到的事件。

if (key.isAcceptable()) {
            handleAccept();
          } else if (key.isReadable()) {
            // deal with reads
            handleRead(key);
          } else if (key.isWritable()) {
            // deal with writes
            handleWrite(key);
          } else {
            LOGGER.warn("Unexpected state in select! " + key.interestOps());
          }

由于TNonblockingServer是用一个SelectAcceptThread处理所有的请求，在并发场景下，其他的IO事件只能等待当前线程依次处理，处理效率略低。
这里记录一下handleRead的处理流程：

protected void handleRead(SelectionKey key) {
  FrameBuffer buffer = (FrameBuffer) key.attachment();
  //将channel中数据读取到buffer中
  if (!buffer.read()) {
    cleanupSelectionKey(key);    
    return;  
  }
  // if the buffer's frame read is complete, invoke the method.
  //判断一帧的数据是否读取完成，读取完成就调用相关的方法，执行rpc调用
  if (buffer.isFrameFullyRead()) {
    if (!requestInvoke(buffer)) {
      cleanupSelectionKey(key);    
      }
  }
}

在buffer.read()中完成实在数据的读取，根据FrameBuffer的状态，分为几部分读取
FrameBufferState.READING_FRAME_SIZE 初始状态，标识读取帧大小，该步完成，状态变更为：FrameBufferState.READING_FRAME
FrameBufferState.READING_FRAME 标识读取实际帧内容，该步完成，状态变更为：FrameBufferState.READ_FRAME_COMPLETE

public boolean read() {
    //读取帧大小
      if (state_ == FrameBufferState.READING_FRAME_SIZE) {
        // try to read the frame size completely
        if (!internalRead()) {
          return false;
        }

        // if the frame size has been read completely, then prepare to read the
        // actual frame.
        if (buffer_.remaining() == 0) {
          // pull out the frame size as an integer.
          int frameSize = buffer_.getInt(0);
          if (frameSize <= 0) {
            LOGGER.error("Read an invalid frame size of " + frameSize
                + ". Are you using TFramedTransport on the client side?");
            return false;
          }

          // if this frame will always be too large for this server, log the
          // error and close the connection.
          if (frameSize > MAX_READ_BUFFER_BYTES) {
            LOGGER.error("Read a frame size of " + frameSize
                + ", which is bigger than the maximum allowable buffer size for ALL connections.");
            return false;
          }

          // if this frame will push us over the memory limit, then return.
          // with luck, more memory will free up the next time around.
          if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
            return true;
          }

          // increment the amount of memory allocated to read buffers
          readBufferBytesAllocated.addAndGet(frameSize + 4);

          // reallocate the readbuffer as a frame-sized buffer
          buffer_ = ByteBuffer.allocate(frameSize + 4);
          buffer_.putInt(frameSize);

          state_ = FrameBufferState.READING_FRAME;
        } else {
          // this skips the check of READING_FRAME state below, since we can't
          // possibly go on to that state if there's data left to be read at
          // this one.
          return true;
        }
      }

      // it is possible to fall through from the READING_FRAME_SIZE section
      // to READING_FRAME if there's already some frame data available once
      // READING_FRAME_SIZE is complete.
      //读取帧大小完成，读取实际的帧数据

      if (state_ == FrameBufferState.READING_FRAME) {
        if (!internalRead()) {
          return false;
        }

        // since we're already in the select loop here for sure, we can just
        // modify our selection key directly.
        if (buffer_.remaining() == 0) {
          // get rid of the read select interests
          selectionKey_.interestOps(0);
          //标识读取帧完成，可以进行下一步
          state_ = FrameBufferState.READ_FRAME_COMPLETE;
        }

        return true;
      }

      // if we fall through to this point, then the state must be invalid.
      LOGGER.error("Read was called but state is invalid (" + state_ + ")");
      return false;
    }

帧读取完成后，调用requestInvoke调用相关的方法，执行RPC调用

/**
     * Actually invoke the method signified by this FrameBuffer.
     */
    public void invoke() {
      frameTrans_.reset(buffer_.array());//设置transport的缓冲区
      response_.reset();//响应缓冲区清零

      try {
        if (eventHandler_ != null) {
          eventHandler_.processContext(context_, inTrans_, outTrans_);
        }
      // 实际调用,通过getProcessor获取服务启动时设置的processor,这个processor是我们通过thrift的接口*.thrift编译生成的类里面的处理类
      processorFactory_.getProcessor(inTrans_).process(inProt_, outProt_);
        //响应结果
        responseReady();
        return;
      } catch (TException te) {
        LOGGER.warn("Exception while invoking!", te);
      } catch (Throwable t) {
        LOGGER.error("Unexpected throwable while invoking!", t);
      }
      // This will only be reached when there is a throwable.
      state_ = FrameBufferState.AWAITING_CLOSE;
      requestSelectInterestChange();
    }

通过 processorFactory_.getProcessor(inTrans_)获取到我们设置的处理类，然后process进行解析处理，首先解析message里面的信息，获取msg，然后通过msg.name获得processMap中对应的处理类，例如echo方法对应的映射关系：
processMap.put(“echo”, new echo());
获取到处理类之后，调用process进行处理，如fn.process(msg.seqid, in, out, iface)

 @Override
  public boolean process(TProtocol in, TProtocol out) throws TException {
    //获取msg
    TMessage msg = in.readMessageBegin();
    //获取实际的处理方法
    ProcessFunction fn = processMap.get(msg.name);
    if (fn == null) {
      TProtocolUtil.skip(in, TType.STRUCT);
      in.readMessageEnd();
      TApplicationException x = new TApplicationException(TApplicationException.UNKNOWN_METHOD, "Invalid method name: '"+msg.name+"'");
      out.writeMessageBegin(new TMessage(msg.name, TMessageType.EXCEPTION, msg.seqid));
      x.write(out);
      out.writeMessageEnd();
      out.getTransport().flush();
      return true;
    }
    //调用处理方法，进行消息处理
    fn.process(msg.seqid, in, out, iface);
    return true;
  }

消息处理的方法见下：

public final void process(int seqid, TProtocol iprot, TProtocol oprot, I iface) throws TException {
    T args = getEmptyArgsInstance();
    try {
      args.read(iprot);
    } catch (TProtocolException e) {
      iprot.readMessageEnd();
      TApplicationException x = new TApplicationException(TApplicationException.PROTOCOL_ERROR, e.getMessage());
      oprot.writeMessageBegin(new TMessage(getMethodName(), TMessageType.EXCEPTION, seqid));
      x.write(oprot);
      oprot.writeMessageEnd();
      oprot.getTransport().flush();
      return;
    }
    iprot.readMessageEnd();
    TBase result = null;

    try {
      //实际的方法调用，会直接调用RPC接口的实际实现
      result = getResult(iface, args);
    } catch(TException tex) {
      LOGGER.error("Internal error processing " + getMethodName(), tex);
      TApplicationException x = new TApplicationException(TApplicationException.INTERNAL_ERROR, 
        "Internal error processing " + getMethodName());
      oprot.writeMessageBegin(new TMessage(getMethodName(), TMessageType.EXCEPTION, seqid));
      x.write(oprot);
      oprot.writeMessageEnd();
      oprot.getTransport().flush();
      return;
    }
    //处理返回结果，通过oprot将结果序列化，有客户端进行解析
    if(!isOneway()) {
      oprot.writeMessageBegin(new TMessage(getMethodName(), TMessageType.REPLY, seqid));
      result.write(oprot);
      oprot.writeMessageEnd();
      oprot.getTransport().flush();
    }
  }

这里对getResult(iface, args)做一下简单描述，以echo接口为例：

 public static class echo<I extends Iface> extends org.apache.thrift.ProcessFunction<I, echo_args> {
      public echo() {
        super("echo");
      }

      public echo_args getEmptyArgsInstance() {
        return new echo_args();
      }

      protected boolean isOneway() {
        return false;
      }

      public echo_result getResult(I iface, echo_args args) throws org.apache.thrift.TException {
        echo_result result = new echo_result();
        //调用echo的实现
        result.success = iface.echo(args.reqId, args.caller, args.srcStr, args.ext);
        return result;
      }
    }

通过iface.echo(args.reqId, args.caller, args.srcStr, args.ext)的调用，我们实现了实际的echo接口的调用，然后通过oprot将返回结果序列化，写入缓冲区。
responseReady方法将结果放入nio的缓冲区，然后通过selectionKey_.interestOps(SelectionKey.OP_WRITE);设置写事件，触发NIO write操作，由handle_write将结果返回给客户端。