zooKeeper使用NIO通信中解决tcp粘包、拆包的方法

Tcp协议是个“流协议”，流就是没有界限的一串数据。Tcp底层并不关心上层业务数据，它会根据Tcp缓冲区的实际情况进行包的划分。所以，在业务上一个完整的数据，可能会被拆分成多个Tcp数据包进行传输，也可能会将业务上的多个数据打包为一个数据包进行传输。所以，必须在Tcp的上层设计应用协议来解决。

业界主流协议的解决方案，可以归纳为如下四种：

1、消息定长。例如每个报文的大小固定为200字节，如果不够，空位补齐；

2、 在数据包尾增加回车换行符进行分割，如FTP协议。

3、 将消息分为消息头和消息体，消息头中包含表示消息总长度（或消息体长度）的字段，通常设计思路为消息头的第一个字段使用int32来表示消息的总长度；

4、更加复杂的设计。

在zooKeeper中，客户端与服务端进行通信，主要采用两种方式：1、原生java NIO接口。2、Netty库。Netty库是对java NIO的封装，使得利用java实现网络通信更加便利。在zooKeeper采用原生java NIO接口实现网络通信时，需要解决Tcp粘包与拆包问题。zooKeeper客户端与服务端通信过程中，采用jute序列化与反序列化上层数据包，其解决tcp粘包与拆包的方案为方案三：添加消息头，消息头只包含表示消息体长度的字段。客户端与服务端通信结构如下代码所示：

/**
     * This class allows us to pass the headers and the relevant records around.
     */
    static class Packet {
        RequestHeader requestHeader;

        ReplyHeader replyHeader;

        Record request;

        Record response;

        ByteBuffer bb;

        /** Client's view of the path (may differ due to chroot) **/
        String clientPath;
        /** Servers's view of the path (may differ due to chroot) **/
        String serverPath;

        boolean finished;

        AsyncCallback cb;

        Object ctx;

        WatchRegistration watchRegistration;

        public boolean readOnly;

        /** Convenience ctor */
        Packet(RequestHeader requestHeader, ReplyHeader replyHeader,
               Record request, Record response,
               WatchRegistration watchRegistration) {
            this(requestHeader, replyHeader, request, response,
                 watchRegistration, false);
        }

        Packet(RequestHeader requestHeader, ReplyHeader replyHeader,
               Record request, Record response,
               WatchRegistration watchRegistration, boolean readOnly) {

            this.requestHeader = requestHeader;
            this.replyHeader = replyHeader;
            this.request = request;
            this.response = response;
            this.readOnly = readOnly;
            this.watchRegistration = watchRegistration;
        }

        public void createBB() {
            try {
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                BinaryOutputArchive boa = BinaryOutputArchive.getArchive(baos);
                boa.writeInt(-1, "len"); // We'll fill this in later
                if (requestHeader != null) {
                	// 将zooKeeper请求头序列化后的字节流写入boa
                    requestHeader.serialize(boa, "header");
                }
                // 将zooKeeper请求体序列化后的字节流写入boa
                if (request instanceof ConnectRequest) {
                    request.serialize(boa, "connect");
                    // append "am-I-allowed-to-be-readonly" flag
                    boa.writeBool(readOnly, "readOnly");
                } else if (request != null) {
                    request.serialize(boa, "request");
                }
                baos.close();
                // 将字节流数组包装为bytebuffer
                this.bb = ByteBuffer.wrap(baos.toByteArray());
                // 在bytebuffer的开始写入zooKeeper请求头+zooKeeper请求体的长度。解决Tcp粘包与拆包
                this.bb.putInt(this.bb.capacity() - 4);
                // position归0
                this.bb.rewind();
            } catch (IOException e) {
                LOG.warn("Ignoring unexpected exception", e);
            }
        }

        @Override
        public String toString() {
            StringBuilder sb = new StringBuilder();

            sb.append("clientPath:" + clientPath);
            sb.append(" serverPath:" + serverPath);
            sb.append(" finished:" + finished);

            sb.append(" header:: " + requestHeader);
            sb.append(" replyHeader:: " + replyHeader);
            sb.append(" request:: " + request);
            sb.append(" response:: " + response);

            // jute toString is horrible, remove unnecessary newlines
            return sb.toString().replaceAll("\r*\n+", " ");
        }
    }

createBB()方法将zooKeeper请求头和zooKeeper请求体进行序列化后包装为ByteBuffer，然后在预占位上写入序列化后的请求长度。当socketChannel可以写入的时候，直接调用socketChannel的write方法即可。

服务端接收到请求的处理代码如下：

          if (k.isReadable()) {
            	// 将内容读取到incomingBuffer中.
            	// 可能读取到lenBuffer中，也可能读取到根据请求内容长度新分配的bytebuffer中
                int rc = sock.read(incomingBuffer);
                if (rc < 0) {
                    throw new EndOfStreamException(
                            "Unable to read additional data from client sessionid 0x"
                            + Long.toHexString(sessionId)
                            + ", likely client has closed socket");
                }
                // 如果读满了
                if (incomingBuffer.remaining() == 0) {
                    boolean isPayload;
                    // 关键：如果incomingBuffer和lenBuffer指向同一个byteBuffer，
                    // 说明是新请求的第一次读,读取的是请求内容长度
                    if (incomingBuffer == lenBuffer) { // start of next request
                        incomingBuffer.flip();
                        // 获取请求内容长度
                        isPayload = readLength(k);
                        incomingBuffer.clear();
                    } else {
                    	// 说明tcp拆包了，需要继续读未读完的包。
                        // continuation
                        isPayload = true;
                    }
                    // 继续读请求内容
                    if (isPayload) { // not the case for 4letterword
                        readPayload();
                    }
                    else {
                        // four letter words take care
                        // need not do anything else
                        return;
                    }
                }
            }

其中成员变量lenBuffer和incomingBuffer的定义为：

ByteBuffer lenBuffer = ByteBuffer.allocate(4);

ByteBuffer incomingBuffer = lenBuffer;

readLength()为读取请求内容长度，代码为：

/** Reads the first 4 bytes of lenBuffer, which could be true length or
     *  four letter word.
     *
     * @param k selection key
     * @return true if length read, otw false (wasn't really the length)
     * @throws IOException if buffer size exceeds maxBuffer size
     */
    private boolean readLength(SelectionKey k) throws IOException {
        // Read the length, now get the buffer
        int len = lenBuffer.getInt();
        if (!initialized && checkFourLetterWord(sk, len)) {
            return false;
        }
        if (len < 0 || len > BinaryInputArchive.maxBuffer) {
            throw new IOException("Len error " + len);
        }
        if (zkServer == null) {
            throw new IOException("ZooKeeperServer not running");
        }
        // 分配请求内容长度大小的ByteBuffer
        incomingBuffer = ByteBuffer.allocate(len);
        return true;
    }

readPayload()为读取请求内容方法，代码为：

 /** Read the request payload (everything following the length prefix) */
    private void readPayload() throws IOException, InterruptedException {
    	// 如果还有剩余空间没有读满，那就接着读。表示tcp拆包了。
        if (incomingBuffer.remaining() != 0) { // have we read length bytes?
            int rc = sock.read(incomingBuffer); // sock is non-blocking, so ok
            if (rc < 0) {
                throw new EndOfStreamException(
                        "Unable to read additional data from client sessionid 0x"
                        + Long.toHexString(sessionId)
                        + ", likely client has closed socket");
            }
        }
        
        // 如果没有剩余空间了，代表已经将请求内容读完整了。
        if (incomingBuffer.remaining() == 0) { // have we read length bytes?
        	// 收到请求包
            packetReceived();
            incomingBuffer.flip();
            if (!initialized) {
                readConnectRequest();
            } else {
                readRequest();
            }
            // 清空
            lenBuffer.clear();
            // 下一个请求包还未读取状态
            incomingBuffer = lenBuffer;
        }
    }

zooKeeper使用固定长度为4字节的lenBuffer来存放请求内容长度，incomingBuffer存放请求内容。如果incomingBuffer==lenBuffer，即两个引用指向预先分配4字节的bytebuffer实例，就代表着新的请求内容长度的读取。否则，表示读取请求内容。其设计很简洁、实用，特此总结，记录。

参考：

1、《Netty权威指南 第二版》

2、 zooKeeper-3.4.6源码