13-ByteBuf

ByteBuf

• ByteBuf 是 Netty 中数据容器，它高效的实现了底层数据通信过程中所需要的字节序列的相关操作。

• ByteBuf 自身的源码就非常多，而且还有很多子类，因此不可能面面俱到，我们分析一些主要的点，对比BIO中的ByteBuffer来对比会更好，ByteBuffer是Java Nio中的字节容器，但是使用上较为复杂，Netty使用ByteBuf来实现了更加强大的功能，我们看看Netty 自己实现的ByteBuf具体有什么优势。

• 考虑到 ByteBuf 的内容非常多，会将其分为几个部分慢慢研究，本文主要关于ByteBuf的API；

一、ByteBuf

• ByteBuf 是一个抽象类，是Netty中具体的ByteBuf的抽象父类,实现了ReferenceCounted 和 Comparable 接口，ReferenceCounted 接口在后面文章分析

public abstract class ByteBuf implements ReferenceCounted, Comparable<ByteBuf> 

1.1 类注释

• 先从类注释获取一些基本的信息,对其有一个大致的了解

/**
 * 一个零个或者多个字节序列，可以随机并且有序访问；此接口提供原始字节的抽象视图
 *
 * A random and sequential accessible sequence of zero or more bytes (octets).
 * This interface provides an abstract view for one or more primitive byte
 * arrays ({@code byte[]}) and {@linkplain ByteBuffer NIO buffers}.
 *
 * <h3>Creation of a buffer</h3>
 *
 * 推荐使用帮助方法Unpooled 来创建一个实例，而不是使用构造方法
 * It is recommended to create a new buffer using the helper methods in
 * {@link Unpooled} rather than calling an individual implementation's
 * constructor.
 *
 * <h3>Random Access Indexing</h3>
 * 可以像数组一样支持随机访问，下标 是: 0 ->  buffer.capacity()-1
 * Just like an ordinary primitive byte array, {@link ByteBuf} uses
 * <a href="http://en.wikipedia.org/wiki/Zero-based_numbering">zero-based indexing</a>.
 * It means the index of the first byte is always {@code 0} and the index of the last byte is
 * always {@link #capacity() capacity - 1}.  For example, to iterate all bytes of a buffer, you
 * can do the following, regardless of its internal implementation:
 *
 * <pre>
 * {@link ByteBuf} buffer = ...;
 * for (int i = 0; i &lt; buffer.capacity(); i ++) {
 *     byte b = buffer.getByte(i);
 *     System.out.println((char) b);
 * }
 * </pre>
 *
 * <h3>Sequential Access Indexing</h3>
 * 提供两个指针变量来支持顺序读写操作，readIndex()用于读操作，writeIndex()用于写操作，
 * 下图展示了buffer如何被两个指针分成三个部分：
 *  0  <=  readIndex <= writeIndex <= capacity
 *  0   丢弃部分 readIndex     可读部分 writeIndex   可写部分 capacity
 *
 * {@link ByteBuf} provides two pointer variables to support sequential
 * read and write operations - {@link #readerIndex() readerIndex} for a read
 * operation and {@link #writerIndex() writerIndex} for a write operation
 * respectively.  The following diagram shows how a buffer is segmented into
 * three areas by the two pointers:
 *
 * <pre>
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      |                   |     (CONTENT)    |                  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 * </pre>
 *
 * <h4>Readable bytes (the actual content)</h4>
 *
 * read开头的操作和skip开头的操作是读取数据，读取数据后readerIndex会自动增加读取的
 * 部分长度，skip会跳过一部分读取；
 *
 * 如果读取操作的参数也是一个ByteBuf对象并且没有指定目标index，那么参数对象的writerIndex也
 * 会增长，如果空间不足，则会抛出：IndexOutOfBoundsException
 *
 * This segment is where the actual data is stored.  Any operation whose name
 * starts with {@code read} or {@code skip} will get or skip the data at the
 * current {@link #readerIndex() readerIndex} and increase it by the number of
 * read bytes.  If the argument of the read operation is also a
 * {@link ByteBuf} and no destination index is specified, the specified
 * buffer's {@link #writerIndex() writerIndex} is increased together.
 * <p>
 * If there's not enough content left, {@link IndexOutOfBoundsException} is
 * raised.  The default value of newly allocated, wrapped or copied buffer's
 * {@link #readerIndex() readerIndex} is {@code 0}.
 *
 * <pre>
 * // Iterates the readable bytes of a buffer.
 * {@link ByteBuf} buffer = ...;
 * while (buffer.isReadable()) {
 *     System.out.println(buffer.readByte());
 * }
 * </pre>
 *
 * <h4>Writable bytes</h4>
 *
 * write开头的操作是写数据，写数据之后writerIndex会增加所写部分的长度
 * 如果写操作的参数也是ByteBuf，并且没有指定index，那么参数对应的ByteBuf的readerIndex也
 * 会一起自增，如果空间不足，则会抛出：IndexOutOfBoundsException
 *
 * 新创建的ByteBuf的writerIndex是0，wrapped的或者copied的Buffer的writerIndex等于capacity
 *
 * This segment is a undefined space which needs to be filled.  Any operation
 * whose name starts with {@code write} will write the data at the current
 * {@link #writerIndex() writerIndex} and increase it by the number of written
 * bytes.  If the argument of the write operation is also a {@link ByteBuf},
 * and no source index is specified, the specified buffer's
 * {@link #readerIndex() readerIndex} is increased together.
 * <p>
 *
 * If there's not enough writable bytes left, {@link IndexOutOfBoundsException}
 * is raised.  The default value of newly allocated buffer's
 * {@link #writerIndex() writerIndex} is {@code 0}.  The default value of
 * wrapped or copied buffer's {@link #writerIndex() writerIndex} is the
 * {@link #capacity() capacity} of the buffer.
 *
 * <pre>
 * // Fills the writable bytes of a buffer with random integers.
 * {@link ByteBuf} buffer = ...;
 * while (buffer.maxWritableBytes() >= 4) {
 *     buffer.writeInt(random.nextInt());
 * }
 * </pre>
 *
 * <h4>Discardable bytes</h4>
 * 已经读取的部分称为丢弃部分，最初是0，随着读取操作的进行，这部分区域越来越大，可以通过
 * discardReadBytes方法来回收该部分的内存区域，调用之后这部分区域会变成可写区域，如下图所示
 *
 * This segment contains the bytes which were read already by a read operation.
 * Initially, the size of this segment is {@code 0}, but its size increases up
 * to the {@link #writerIndex() writerIndex} as read operations are executed.
 * The read bytes can be discarded by calling {@link #discardReadBytes()} to
 * reclaim unused area as depicted by the following diagram:
 *
 * <pre>
 *  BEFORE discardReadBytes()
 *
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 *
 *
 *  AFTER discardReadBytes()
 *
 *      +------------------+--------------------------------------+
 *      |  readable bytes  |    writable bytes (got more space)   |
 *      +------------------+--------------------------------------+
 *      |                  |                                      |
 * readerIndex (0) <= writerIndex (decreased)        <=        capacity
 * </pre>
 *
 * 注意writable bytes是不能保证的，多数情况下不会被移动，这取决的底层实现
 * Please note that there is no guarantee about the content of writable bytes
 * after calling {@link #discardReadBytes()}.  The writable bytes will not be
 * moved in most cases and could even be filled with completely different data
 * depending on the underlying buffer implementation.
 *
 * <h4>Clearing the buffer indexes</h4>
 *
 * clear方法可以将读写索引置零，但是并不会清楚buffer的内容，注意clear操作的语义
 * 和ByteBuffer#clear()是不一样的
 *
 * You can set both {@link #readerIndex() readerIndex} and
 * {@link #writerIndex() writerIndex} to {@code 0} by calling {@link #clear()}.
 * It does not clear the buffer content (e.g. filling with {@code 0}) but just
 * clears the two pointers.  Please also note that the semantic of this
 * operation is different from {@link ByteBuffer#clear()}.
 *
 * <pre>
 *  BEFORE clear()
 *
 *      +-------------------+------------------+------------------+
 *      | discardable bytes |  readable bytes  |  writable bytes  |
 *      +-------------------+------------------+------------------+
 *      |                   |                  |                  |
 *      0      <=      readerIndex   <=   writerIndex    <=    capacity
 *
 *
 *  AFTER clear()
 *
 *      +---------------------------------------------------------+
 *      |             writable bytes (got more space)             |
 *      +---------------------------------------------------------+
 *      |                                                         |
 *      0 = readerIndex = writerIndex            <=            capacity
 * </pre>
 *
 * <h3>Search operations</h3>
 * 使用indexOf，bytesBefore，forEachByte来搜索
 * For simple single-byte searches, use {@link #indexOf(int, int, byte)} and {@link #bytesBefore(int, int, byte)}.
 * {@link #bytesBefore(byte)} is especially useful when you deal with a {@code NUL}-terminated string.
 * For complicated searches, use {@link #forEachByte(int, int, ByteProcessor)} with a {@link ByteProcessor}
 * implementation.
 *
 * <h3>Mark and reset</h3>
 *  markReaderIndex 和 markWriterIndex分表标记读写索引的位置
 * There are two marker indexes in every buffer. One is for storing
 * {@link #readerIndex() readerIndex} and the other is for storing
 * {@link #writerIndex() writerIndex}.  You can always reposition one of the
 * two indexes by calling a reset method.  It works in a similar fashion to
 * the mark and reset methods in {@link InputStream} except that there's no
 * {@code readlimit}.
 *
 * <h3>Derived buffers</h3>
 * 派生缓冲区，下面方法会给一个已存在的buffer创建一个缓冲区视图，(内部存储的是一份数据，改动是关联
 * 的，使用一个引用计数，但是读写索引和索引标记是相互独立的)
 *
 * 如果需要一份独立的拷贝，使用copy方法
 * You can create a view of an existing buffer by calling one of the following methods:
 * <ul>
 *   <li>{@link #duplicate()}</li>
 *   <li>{@link #slice()}</li>
 *   <li>{@link #slice(int, int)}</li>
 *   <li>{@link #readSlice(int)}</li>
 *   <li>{@link #retainedDuplicate()}</li>
 *   <li>{@link #retainedSlice()}</li>
 *   <li>{@link #retainedSlice(int, int)}</li>
 *   <li>{@link #readRetainedSlice(int)}</li>
 * </ul>
 * A derived buffer will have an independent {@link #readerIndex() readerIndex},
 * {@link #writerIndex() writerIndex} and marker indexes, while it shares
 * other internal data representation, just like a NIO buffer does.
 * <p>
 * In case a completely fresh copy of an existing buffer is required, please
 * call {@link #copy()} method instead.
 *
 * <h4>Non-retained and retained derived buffers</h4>
 *
 * 注意duplicate、slice、slice、readSlice这些方法返回派生缓冲区是不会调用retain增加引用计数器的，
 *
 * 如果需要创建一个buffer并且增加引用计数器，那么使用retainedDuplicate、retainedSlice、retainedSlice、readRetainedSlice，
 * 它们返回的buffer会产生更少的垃圾
 *
 * Note that the {@link #duplicate()}, {@link #slice()}, {@link #slice(int, int)} and {@link #readSlice(int)} does NOT
 * call {@link #retain()} on the returned derived buffer, and thus its reference count will NOT be increased. If you
 * need to create a derived buffer with increased reference count, consider using {@link #retainedDuplicate()},
 * {@link #retainedSlice()}, {@link #retainedSlice(int, int)} and {@link #readRetainedSlice(int)} which may return
 * a buffer implementation that produces less garbage.
 *
 * <h3>Conversion to existing JDK types</h3>
 *
 * <h4>Byte array</h4>
 *
 * 如果ByteBuf内部使用byte array来保存数据，那么可以通过array方法来获取这个字节数组，
 * 使用hasArray可以判断其是不是由byte array来保存数据的
 * If a {@link ByteBuf} is backed by a byte array (i.e. {@code byte[]}),
 * you can access it directly via the {@link #array()} method.  To determine
 * if a buffer is backed by a byte array, {@link #hasArray()} should be used.
 *
 * <h4>NIO Buffers</h4>
 * 如果一个ByteBuf能够转换为一个NIO的ByteBuffer，并且内容共享，可以调用nioBuffer方法来做转换，
 * 使用nioBufferCount判断一个ByteBuf 是否可以转换为一个 NIO的ByteBuffer
 * If a {@link ByteBuf} can be converted into an NIO {@link ByteBuffer} which shares its
 * content (i.e. view buffer), you can get it via the {@link #nioBuffer()} method.  To determine
 * if a buffer can be converted into an NIO buffer, use {@link #nioBufferCount()}.
 *
 * <h4>Strings</h4>
 *
 * Various {@link #toString(Charset)} methods convert a {@link ByteBuf}
 * into a {@link String}.  Please note that {@link #toString()} is not a
 * conversion method.
 *
 * <h4>I/O Streams</h4>
 *
 * Please refer to {@link ByteBufInputStream} and
 * {@link ByteBufOutputStream}.
 */

• 小结如下：

1.ByteBuf内部是字节序列，支持随机访问和顺序访问，可以通过下标访问，使用Unpooled创建
2.内部通过读写两个索引来控制读写，两个索引都要对应的标记方法，读写需要注意越界异常
3.已经读取过的内容相当于废弃内容，可以通过discardReadBytes来腾出该部分空间用于写入，不过这取决于不同子类实现
4.派生缓冲区会共享数据，但是读写索引是独立的，引用计数是共享的，对应的也有一些方法提供buffer并且增加引用计数
5.可以通过方法获取内部的字节数组

1.2 方法

• 方法非常多，只能归类熟悉

1.2.1 属性相关

• ByteBuf 基本属性相关的方法

public abstract int capacity(); // 容量
public abstract ByteBuf capacity(int newCapacity);
public abstract int maxCapacity(); // 最大容量

public abstract ByteBufAllocator alloc(); // 分配器，用于创建 ByteBuf 对象。

public abstract ByteBuf unwrap(); // 获得被包装( wrap )的 ByteBuf 对象。

public abstract boolean isDirect(); // 是否 NIO Direct Buffer

public abstract boolean isReadOnly(); // 是否为只读 Buffer

public abstract int readerIndex(); //获取读取位置
public abstract ByteBuf readerIndex(int readerIndex); //设置读取位置
public abstract int writerIndex(); //获取写入位置
public abstract ByteBuf writerIndex(int writerIndex); //设置写入位置
public abstract ByteBuf setIndex(int readerIndex, int writerIndex); // 设置读取和写入位置
public abstract int readableBytes(); // 剩余可读字节数
public abstract int writableBytes(); // 剩余可写字节数
public abstract ByteBuf markReaderIndex(); // 标记读取位置
public abstract ByteBuf markWriterIndex(); // 标记写入位置

1.2.2 读写数据

• 数据读取方法，可以按照不同的类型读取，读取的字节数也不一样,下面只给出Int类型的读写，其他类型的读写类似

// Int 4 字节
//读取
//绝对读，不改变索引
public abstract int   getInt(int index); //指定位置读取int，不改变读索引
public abstract int   getIntLE(int index); //指定位置读取int，不改变读索引，以小端的方式读取

//普通读，改变索引
public abstract int   readInt();//读取int，readerIndex增加4
public abstract int   readIntLE();//读取int，readerIndex增加4，以小端的方式读取


//写入
//绝对写，不改变索引
public abstract ByteBuf setInt(int index, int value);//在指定位置写入int，不改变读写索引
public abstract ByteBuf setIntLE(int index, int value);//在指定位置写入int，不改变读写索引，以小端的方式写入

//普通写，改变索引
public abstract ByteBuf writeInt(int value);//写入int，并且增加写索引
public abstract ByteBuf writeIntLE(int value);//写入int，并且增加写索引

• 这里还有很多其他类型的方法没有给出，比如读取无符号数等，但是讨论都一样，我们看到读和写都有两类方法；直接通过get和set的读写方法是不改变索引的，因为是随机访问直接通过index操作，而read/write对应的读写API的会改变索引的，相当于在前面索引的基础之上再继续操作；
• 其他还有很多类型，各种基本类型、数组、Medium 类型等操作都有类似的API

1.2.3 查找方法

// 指定值( value ) 在 ByteBuf 中的位置，包左不包右
public abstract int indexOf(int fromIndex, int toIndex, byte value); 


//查找第一次出现value的位置
public abstract int bytesBefore(byte value); 


//从读索引开始查找length长度，查找value出现的位置，不改变读写索引
public abstract int bytesBefore(int length, byte value); 


//从index开始查找length的长度，查找value出现的位置，不改变读写索引
public abstract int bytesBefore(int index, int length, byte value);


// 遍历 ByteBuf ，进行自定义处理
public abstract int forEachByte(ByteProcessor processor); 
public abstract int forEachByte(int index, int length, ByteProcessor processor);
//倒序遍历处理
public abstract int forEachByteDesc(ByteProcessor processor);
public abstract int forEachByteDesc(int index, int length, ByteProcessor processor);

1.2.4 释放操作

public abstract ByteBuf discardReadBytes(); // 释放已读的字节空间，参考前面的注释里面
public abstract ByteBuf discardSomeReadBytes(); // 释放部分已读的字节空间，可能会释放也可能不释放
public abstract ByteBuf clear(); // 清空字节空间。实际是修改 readerIndex=writerIndex=0，标记清空。

• discardReadBytes 方法会将剩余的可读部分移动到字节数组的头部，由此腾出空间放在最后面写入，这是一个时间换空间的操作，在AbstractByteBuf的实现如下：

@Override
    public ByteBuf discardReadBytes() {
        //1.校验可访问
        ensureAccessible();
        //2.无废弃段，则直接返回
        if (readerIndex == 0) {
            return this;
        }

        //3.未读取完
        if (readerIndex != writerIndex) {
            //4.将可读段复制到ByteBuf头部空间，由子类实现
            setBytes(0, this, readerIndex, writerIndex - readerIndex);
            //5.写索引减小
            writerIndex -= readerIndex;
            //6.调整标记位
            adjustMarkers(readerIndex);
            //7.读索引重置为 0
            readerIndex = 0;
        } else {
            //8.全部读取完，则调整标记位
            adjustMarkers(readerIndex);
            //9.索引重置
            writerIndex = readerIndex = 0;
        }
        //10.返回当前对象
        return this;
    }

• 在AbstractByteBuf 中discardSomeReadBytes方法会判断废弃的内容是否超过总空间的一半，超过则回收，反之不会回收

1.2.5 拷贝操作

• 按照前面类注释的说明，拷贝方法有两类，一类是拷贝后的内容是使用同一份的，相互关联的，但是读写索引是独立的，这种也叫派生buffer，注意这里面有些方法不会自增引用计数，带retained的方法，会自增引用计数

public abstract ByteBuf duplicate(); // 拷贝整个的字节数组。共享，相互影响。
public abstract ByteBuf slice(); // 拷贝可读部分的字节数组。共享，相互影响。
public abstract ByteBuf slice(int index, int length);

public abstract ByteBuf readSlice(int length);
public abstract ByteBuf readRetainedSlice(int length);

public abstract ByteBuf retainedSlice();
public abstract ByteBuf retainedSlice(int index, int length);
public abstract ByteBuf retainedDuplicate();

• 另一类是内容一致，但是相互独立，主要是copy 方法

public abstract ByteBuf copy(); // 拷贝可读部分的字节数组。独立，互相不影响。
public abstract ByteBuf copy(int index, int length);

1.2.6 其他

• 另外包括NIO转换方法，在类注释中说明了，主要是：nioBuffer() 和 nioBufferCount() 方法
• 还有获取内部的字节数组，主要是：array() 方法和 hasArray() 方法，适用于Heap类型的 ByteBuf

二、ByteBuf和ByteBuffer对比

• 主要对比一下ByteBuf 和NIO 的byteBuffer 之间的区别

1.读和写使用了不同的索引，API更友好，因此也不需要flip 切换读写模式
2.容量可以按需增长，ByteBuffer 不能
3.支持引用计数，因此回收更及时，更灵活
4.支持池化，资源复用，支持零拷贝，提高性能
5.支持方法的链式调用，API更友好

• ByteBuffer 为什么不能扩容:如下所示其内部的hb数组是final类型的，因此在创建之后不能修改，因此无法扩容

public abstract class ByteBuffer extends Buffer implements Comparable<ByteBuffer>{

    final byte[] hb;                  // Non-null only for heap buffers
    final int offset;
    boolean isReadOnly;                 // Valid only for heap buffers

• 后续文章继续分析ByteBuf 的一些特性，比如子类实现、扩容机制、分配方式等；