Netty源码(十二)之服务端向客户端写数据的过程

前面的几篇博客大致的介绍整个Netty核心的代码，以及启动的流程，服务端的读取流程，以及常用的解码器的源码。至此Netty的源码还剩一个服务端向客户端写数据的流程。今天我们就来介绍一下服务端向客户端写的流程。写的方式有以下两种，具体代码如下：

ctx.writeAndFlush("");
ctx.channel().writeAndFlush("");

这两种方式的区别具体如下图所示：

可以看到我们如果执行ctx.writeAndFlush("");方法后，当前的handler后面所有handler都会忽略。如果是执行ctx.channel().writeAndFlush("");就会从最后一个handler往前执行。

本文讲的是ctx.channel().writeAndFlush("");废话不多说，我们直接看代码，具体的代码如下：

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
  @Override
  public ChannelFuture writeAndFlush(Object msg) {
    //方法调用
    return pipeline.writeAndFlush(msg);
  }
}

public class DefaultChannelPipeline implements ChannelPipeline {
  @Override
  public final ChannelFuture writeAndFlush(Object msg) {
    //调用尾节点的对应的方法
    return tail.writeAndFlush(msg);
  }
}

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
  @Override
  public ChannelFuture writeAndFlush(Object msg) {
    //继续调用
    return writeAndFlush(msg, newPromise());
  }
  
  @Override
  public ChannelFuture writeAndFlush(Object msg, ChannelPromise promise) {
    //继续调用
    write(msg, true, promise);
    return promise;
  }
  
  private void write(Object msg, boolean flush, ChannelPromise promise) {
    ObjectUtil.checkNotNull(msg, "msg");
    try {
      if (isNotValidPromise(promise, true)) {
        ReferenceCountUtil.release(msg);
        // cancelled
        return;
      }
    } catch (RuntimeException e) {
      ReferenceCountUtil.release(msg);
      throw e;
    }
    final AbstractChannelHandlerContext next = findContextOutbound(flush ?
                (MASK_WRITE | MASK_FLUSH) : MASK_WRITE);
    final Object m = pipeline.touch(msg, next);
    EventExecutor executor = next.executor();
    if (executor.inEventLoop()) {
      if (flush) {
        next.invokeWriteAndFlush(m, promise);
      } else {
        next.invokeWrite(m, promise);
      }
    } else {
      final AbstractWriteTask task;
      if (flush) {
        task = WriteAndFlushTask.newInstance(next, m, promise);
      } else {
        task = WriteTask.newInstance(next, m, promise);
      }
      if (!safeExecute(executor, task, promise, m)) {
        // We failed to submit the AbstractWriteTask. We need to cancel it so we decrement the pending bytes
        // and put it back in the Recycler for re-use later.
        //
        // See https://github.com/netty/netty/issues/8343.
        task.cancel();
      }
    }
  }
}

上面的代码走来进行了一些检验，看看是否有错误，然后就是从尾节点开始找上一个为出栈处理器，然后会调用它的invokeWriteAndFlush(m, promise);的方法，假设这儿我们没有添加出栈任何处理，这儿就会直接调用到头节点(HeadContext)中的invokeWriteAndFlush的方法。我们继续跟进对应的代码，具体的代码如下：

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
  private void invokeWriteAndFlush(Object msg, ChannelPromise promise) {
    if (invokeHandler()) {
      invokeWrite0(msg, promise);
      invokeFlush0();
    } else {
      writeAndFlush(msg, promise);
    }
  }
}

上面的代码走来就是调用对应invokeHandler()方法进行判断，当我们打开对应的代码，发现这个方法主要判断这个节点是否添加成功。很明显这个头节点是添加成功的。于是就会调用invokeWrite0(msg, promise);方法，具体的代码如下：

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
  private void invokeWrite0(Object msg, ChannelPromise promise) {
    try {
      ((ChannelOutboundHandler) handler()).write(this, msg, promise);
    } catch (Throwable t) {
      notifyOutboundHandlerException(t, promise);
    }
  }
}

public class DefaultChannelPipeline implements ChannelPipeline {
  final class HeadContext extends AbstractChannelHandlerContext
    implements ChannelOutboundHandler, ChannelInboundHandler {
    @Override
    public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
      unsafe.write(msg, promise);
    }
  }
}

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
  @Override
  public final void write(Object msg, ChannelPromise promise) {
    assertEventLoop();
    ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
    if (outboundBuffer == null) {
      // If the outboundBuffer is null we know the channel was closed and so
      // need to fail the future right away. If it is not null the handling of the rest
      // will be done in flush0()
      // See https://github.com/netty/netty/issues/2362
      safeSetFailure(promise, newClosedChannelException(initialCloseCause));
      // release message now to prevent resource-leak
      ReferenceCountUtil.release(msg);
      return;
    }
    int size;
    try {
      msg = filterOutboundMessage(msg);
      size = pipeline.estimatorHandle().size(msg);
      if (size < 0) {
        size = 0;
      }
    } catch (Throwable t) {
      safeSetFailure(promise, t);
      ReferenceCountUtil.release(msg);
      return;
    }
    outboundBuffer.addMessage(msg, size, promise);
  }
}

走来先判断outboundBuffer是否为空，这儿由于在类的初始化的时候，这个属性就已经赋值，所以这儿不为空。我们直接看下面的filterOutboundMessage(msg);，对应的代码如下：

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
  @Override
  protected final Object filterOutboundMessage(Object msg) {
    if (msg instanceof ByteBuf) {
      ByteBuf buf = (ByteBuf) msg;
      if (buf.isDirect()) {
        return msg;
      }
      return newDirectBuffer(buf);
    }
    if (msg instanceof FileRegion) {
      return msg;
    }
    throw new UnsupportedOperationException(
      "unsupported message type: " + StringUtil.simpleClassName(msg) + EXPECTED_TYPES);
  }
}

可以看到上面的代码是判断传输的过程中ByteBuf是否是属于堆外内存，如果不是，将原来的ByteBuf转成堆外内存上的ByteBuf，并将原来的ByteBuf进行释放，具体的代码在newDirectBuffer(buf);中实现的。这个时候我们需要返回到原来的代码调用的地方，具体的代码如下：

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
  @Override
  public final void write(Object msg, ChannelPromise promise) {
    assertEventLoop();
    ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
    if (outboundBuffer == null) {
      // If the outboundBuffer is null we know the channel was closed and so
      // need to fail the future right away. If it is not null the handling of the rest
      // will be done in flush0()
      // See https://github.com/netty/netty/issues/2362
      safeSetFailure(promise, newClosedChannelException(initialCloseCause));
      // release message now to prevent resource-leak
      ReferenceCountUtil.release(msg);
      return;
    }
    int size;
    try {
      //转换成堆外内存
      msg = filterOutboundMessage(msg);
      size = pipeline.estimatorHandle().size(msg);
      if (size < 0) {
        size = 0;
      }
    } catch (Throwable t) {
      safeSetFailure(promise, t);
      ReferenceCountUtil.release(msg);
      return;
    }
    outboundBuffer.addMessage(msg, size, promise);
  }
}

然后执行pipeline.estimatorHandle().size(msg);方法，这个方法主要是计算msg的字节长度，具体的代码如下：

public final class DefaultMessageSizeEstimator implements MessageSizeEstimator {
  private static final class HandleImpl implements Handle {
    @Override
    public int size(Object msg) {
      if (msg instanceof ByteBuf) {
        return ((ByteBuf) msg).readableBytes();
      }
      if (msg instanceof ByteBufHolder) {
        return ((ByteBufHolder) msg).content().readableBytes();
      }
      if (msg instanceof FileRegion) {
        return 0;
      }
      return unknownSize;
    }
  }
}

这个时候原来的write方法中还有最后一个调用outboundBuffer.addMessage(msg, size, promise);具体的代码如下：

public final class ChannelOutboundBuffer {
  public void addMessage(Object msg, int size, ChannelPromise promise) {
    Entry entry = Entry.newInstance(msg, size, total(msg), promise);
    if (tailEntry == null) {
      flushedEntry = null;
    } else {
      Entry tail = tailEntry;
      tail.next = entry;
    }
    tailEntry = entry;
    if (unflushedEntry == null) {
      unflushedEntry = entry;
    }
    // increment pending bytes after adding message to the unflushed arrays.
    // See https://github.com/netty/netty/issues/1619
    incrementPendingOutboundBytes(entry.pendingSize, false);
  }
}

走来将传进来的msg封装成一个Entry对象。由于上面的代码，用文字叙述比较难，我们还是画图表示吧！具体的图如下：

可以看到随着每次写入的数据的增加，tailEntry永远指向最后一个，unflushedEntry永远指向第一个没有flush的Entry,而flushedEntry永远指向null。他们维护一个如上图的结构。最后执行incrementPendingOutboundBytes(entry.pendingSize, false);方法，我们打开该方法，具体的代码如下：

public final class ChannelOutboundBuffer {
  private void incrementPendingOutboundBytes(long size, boolean invokeLater) {
    if (size == 0) {
      return;
    }
    long newWriteBufferSize = TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, size);
    if (newWriteBufferSize > channel.config().getWriteBufferHighWaterMark()) {
      setUnwritable(invokeLater);
    }
  }
}

可以发现将这个字节的长度累加到totalPendingSize变量中，最后又一个判读，就是能缓存到最大长度是多少？于是跟随源码，找到如下的答案，具体如下：

private static final int DEFAULT_HIGH_WATER_MARK = 64 * 1024;

也是说这儿最大的缓存的是64K，这儿会调用setUnwritable(invokeLater);方法，具体的代码如下：

public final class ChannelOutboundBuffer {
  private void setUnwritable(boolean invokeLater) {
    for (;;) {
      final int oldValue = unwritable;
      final int newValue = oldValue | 1;
      if (UNWRITABLE_UPDATER.compareAndSet(this, oldValue, newValue)) {
        if (oldValue == 0 && newValue != 0) {
          fireChannelWritabilityChanged(invokeLater);
        }
        break;
      }
    }
  }
}

可以发现上面的代码通过自旋的方法，将unwritable的状态从0改成1，也就是不可写的状态，同时会调用channelWritabilityChanged方法。至此整个invokeWrite0(msg, promise);方法就执行完了。我们再次回到原来代码的调用的地方。具体如下：

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
  private void invokeWriteAndFlush(Object msg, ChannelPromise promise) {
    if (invokeHandler()) {
      invokeWrite0(msg, promise);
      invokeFlush0();
    } else {
      writeAndFlush(msg, promise);
    }
  }
}

这个时候会继续执行invokeFlush0();方法，具体的代码如下：

abstract class AbstractChannelHandlerContext implements ChannelHandlerContext, ResourceLeakHint {
  private void invokeFlush0() {
    try {
      ((ChannelOutboundHandler) handler()).flush(this);
    } catch (Throwable t) {
      notifyHandlerException(t);
    }
  }
}

public class DefaultChannelPipeline implements ChannelPipeline {
  final class HeadContext extends AbstractChannelHandlerContext
    implements ChannelOutboundHandler, ChannelInboundHandler {
    @Override
    public void flush(ChannelHandlerContext ctx) {
      unsafe.flush();
    }
  }
}

public abstract class AbstractChannel extends DefaultAttributeMap implements Channel {
  @Override
  public final void flush() {
    assertEventLoop();
    ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
    if (outboundBuffer == null) {
      return;
    }
    outboundBuffer.addFlush();
    flush0();
  }
}

上面的代码进过一连串的调用最终调用outboundBuffer.addFlush();方法，具体的代码如下：

public final class ChannelOutboundBuffer {
  public void addFlush() {
    // There is no need to process all entries if there was already a flush before and no new messages
    // where added in the meantime.
    //
    // See https://github.com/netty/netty/issues/2577
    Entry entry = unflushedEntry;
    if (entry != null) {
      if (flushedEntry == null) {
        // there is no flushedEntry yet, so start with the entry
        flushedEntry = entry;
      }
      do {
        flushed ++;
        if (!entry.promise.setUncancellable()) {
          // Was cancelled so make sure we free up memory and notify about the freed bytes
          //被刷新的ByteBuf的长度
          int pending = entry.cancel();
          decrementPendingOutboundBytes(pending, false, true);
        }
        entry = entry.next;
      } while (entry != null);
      // All flushed so reset unflushedEntry
      unflushedEntry = null;
    }
  }
}

上面的代码具体的操作如下图所示：

说到底就是几个指针来回的操作。每循环一次调用decrementPendingOutboundBytes(pending, false, true);方法，将原来totalPendingSize中的值减去。最后这个方法就执行完了。最后回到原来的调用的方法的地方，发现还调用了flush0();方法。具体的代码如下：

protected void flush0() {
  if (inFlush0) {
    // Avoid re-entrance
    return;
  }
  final ChannelOutboundBuffer outboundBuffer = this.outboundBuffer;
  if (outboundBuffer == null || outboundBuffer.isEmpty()) {
    return;
  }
  inFlush0 = true;
  // Mark all pending write requests as failure if the channel is inactive.
  if (!isActive()) {
    try {
      if (isOpen()) {
        outboundBuffer.failFlushed(new NotYetConnectedException(), true);
      } else {
        // Do not trigger channelWritabilityChanged because the channel is closed already.
        outboundBuffer.failFlushed(newClosedChannelException(initialCloseCause), false);
      }
    } finally {
      inFlush0 = false;
    }
    return;
  }
  try {
    doWrite(outboundBuffer);
  } catch (Throwable t) {
    if (t instanceof IOException && config().isAutoClose()) {
      initialCloseCause = t;
      close(voidPromise(), t, newClosedChannelException(t), false);
    } else {
      try {
        shutdownOutput(voidPromise(), t);
      } catch (Throwable t2) {
        initialCloseCause = t;
        close(voidPromise(), t2, newClosedChannelException(t), false);
      }
    }
  } finally {
    inFlush0 = false;
  }
}

上面的代码最核心的方法就是doWrite(outboundBuffer);方法，其他的都是一些的异常的处理，我们具体看下这个方法的代码，具体的代码如下：

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
  @Override
  protected void doWrite(ChannelOutboundBuffer in) throws Exception {
    //这个值是16
    int writeSpinCount = config().getWriteSpinCount();
    do {
      //获取flushedEntry的值
      Object msg = in.current();
      if (msg == null) {
        // Wrote all messages.
        clearOpWrite();
        // Directly return here so incompleteWrite(...) is not called.
        return;
      }
      writeSpinCount -= doWriteInternal(in, msg);
    } while (writeSpinCount > 0);
    incompleteWrite(writeSpinCount < 0);
  }
}

上面的代码会调用doWriteInternal(in, msg);方法，具体的代码如下：

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
  private int doWriteInternal(ChannelOutboundBuffer in, Object msg) throws Exception {
    if (msg instanceof ByteBuf) {
      ByteBuf buf = (ByteBuf) msg;
      //如果不可读直接返回0，并移除ChannelOutboundBuffer
      if (!buf.isReadable()) {
        in.remove();
        return 0;
      }
      //调用doWriteBytes(buf);
      final int localFlushedAmount = doWriteBytes(buf);
      if (localFlushedAmount > 0) {
        in.progress(localFlushedAmount);
        //移除当前的entry
        if (!buf.isReadable()) {
          in.remove();
        }
        return 1;
      }
    } else if (msg instanceof FileRegion) {
      FileRegion region = (FileRegion) msg;
      if (region.transferred() >= region.count()) {
        in.remove();
        return 0;
      }
      long localFlushedAmount = doWriteFileRegion(region);
      if (localFlushedAmount > 0) {
        in.progress(localFlushedAmount);
        if (region.transferred() >= region.count()) {
          in.remove();
        }
        return 1;
      }
    } else {
      // Should not reach here.
      throw new Error();
    }
    return WRITE_STATUS_SNDBUF_FULL;
  }
}

上面的代码走来判断取出来的Entry中buf是否是可读的，如果是不可读将当前Entry中链表中移除，然后返回0。如果这个buf是可以读的，就会调用下面的doWriteBytes(buf);方法，具体的代码如下：

@Override
protected int doWriteBytes(final ByteBuf byteBuf) throws Exception {
  //取出读指针
  final int expectedWrittenBytes = byteBuf.readableBytes();
 	//写入到客户端
  return byteBuf.readBytes(javaChannel(), expectedWrittenBytes);
}

上面的代码将buf中内容写入到客户端，并返回写入到通道中的字节数。写入成功后直接返回1，如果写入失败直接返回int的最大值。这个时候写完读指针也是会改变的，这个返回会直接调用in.remove()方法将当前的entry删除。我们再次返回到原来的方法的调用的地方，具体的代码如下：

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
  @Override
  protected void doWrite(ChannelOutboundBuffer in) throws Exception {
    //这个值是16
    int writeSpinCount = config().getWriteSpinCount();
    do {
      //获取flushedEntry的值
      Object msg = in.current();
      if (msg == null) {
        // Wrote all messages.
        clearOpWrite();
        // Directly return here so incompleteWrite(...) is not called.
        return;
      }
      writeSpinCount -= doWriteInternal(in, msg);
    } while (writeSpinCount > 0);
    incompleteWrite(writeSpinCount < 0);
  }
}

这个时候会将writeSpinCount变量减去每次返回的值，一旦发生了写失败，会直接结束这个循环。如果没有发生写失败，就会执行16次这个循环，最后执行 incompleteWrite(writeSpinCount < 0);方法，这个方法中传入writeSpinCount < 0，具体的代码如下：

public abstract class AbstractNioByteChannel extends AbstractNioChannel {
  protected final void incompleteWrite(boolean setOpWrite) {
    // Did not write completely.
    if (setOpWrite) {
      setOpWrite();
    } else {
      clearOpWrite();
      // Schedule flush again later so other tasks can be picked up in the meantime
      eventLoop().execute(flushTask);
    }
  }
  
  protected final void setOpWrite() {
    final SelectionKey key = selectionKey();
    if (!key.isValid()) {
      return;
    }
    final int interestOps = key.interestOps();
    if ((interestOps & SelectionKey.OP_WRITE) == 0) {
      key.interestOps(interestOps | SelectionKey.OP_WRITE);
    }
  }
}

由于writeSpinCount < 0这个参数为true的情况只有两种：一种是写入失败，一种是写入了16次。这个时候这个方法最终会给客户端注册一个SelectionKey.OP_WRITE事件。这个参数为false的时候，只有一种情况，就是msg == null的时候，就会取消SelectionKey.OP_WRITE事件。最后关于NIO SelectionKey事件理解可以参考这篇博客。