kafka源码解析（6）kafka网络设计如何用nio建立连接发送数据

kafka网络设计发送流程，较为复杂，画了一个全局图后才算明了，下面解读下

sendproducerData作用是封装消息，配置新channel,消息放入kafkaChannel，注册读写连接事件
poll方法才是真的用nio组件进行网络发送，selector处理这些事件

sendProducerData流程

send的发送数据方法sendProducerData中，首先会调用recordAccumulator的ready（红色）方法检查哪个消息分区可以发送数据得到ReadyCheckResult（包含可以发送的tp对应的node），顺便看下有没有没见过的新topicPartition,有后面就去更新元数据，然后调用networkClient的ready方法，这就是要检查node的连接状态了，没联通的node从ReadyCheckResult相关信息中去掉。最后封装request并发送的方法sendProduceRequests

sendProducerData : networkClient.ready（绿色）

ready方法如下所示, isReady会判断：当源数据不需要更新且网络连接ready且channel ready且inFlightRequests没达到最大时候，就判定node准备改好了。代码再向下走，就判定下node是否不存在，不存在就initiateConnect初始化连接。

public boolean ready(Node node, long now) {
        if (node.isEmpty())
            throw new IllegalArgumentException("Cannot connect to empty node " + node);

        if (isReady(node, now))
            return true;

        if (connectionStates.canConnect(node.idString(), now))
            // if we are interested in sending to a node and we don't have a connection to it, initiate one
            initiateConnect(node, now);

        return false;
    }

initiateConnect主要是做这事

	selector.connect(nodeConnectionId,
    new InetSocketAddress(address, node.port()),
    this.socketSendBuffer,
    this.socketReceiveBuffer);

selecter里面封装了java.nio.selecter,nio.selecter是运行单线程处理多个 Channel的类，如果应用打开了多个通道，但每个连接的流量都很低，使用 Selector 就会很方便。例如在一个聊天服务器中。使用Selector, 得向 Selector 注册 Channel，然后调用它的 select()方法。这个方法会一直阻塞到某个注册的通道有事件就绪。一旦这个方法返回，线程就可以处理这些事件，
事件的例子有如新的连接进来、数据接收等。

connect里面都是nio。selecter的常规操作，新建socketChannel,配置socketChannel（缓冲区大小，非阻塞模式，关闭nagle算法，不采用小数据包合并）。向selector注册连接事件OP_CONNECT。

registerChannel里面用buildAndAttachKafkaChannel方法把这个连接事件和这个kafkaChannel做了绑定。kafkaChannel和主机也做了kv连接写入map方便查找。

如果connected连上了，就把注册的事件删除。

connect方法建立了kafkaChannel并且做了配置，和node id做了绑定。

public void connect(String id, InetSocketAddress address, int sendBufferSize, int receiveBufferSize) throws IOException {
        ensureNotRegistered(id);
        SocketChannel socketChannel = SocketChannel.open();
        try {
            configureSocketChannel(socketChannel, sendBufferSize, receiveBufferSize);
            //因为是非阻塞模式， boolean connected立即返回，不一定会是true
            boolean connected = doConnect(socketChannel, address);
            SelectionKey key = registerChannel(id, socketChannel, SelectionKey.OP_CONNECT);

            if (connected) {
                // OP_CONNECT won't trigger for immediately connected channels
                log.debug("Immediately connected to node {}", id);
                immediatelyConnectedKeys.add(key);
                key.interestOps(0);
            }
        } catch (IOException | RuntimeException e) {
            socketChannel.close();
            throw e;
        }
    }

networkClient poll流程

客户端的poll方法核心是selector的poll方法

selector的poll方法

public void poll(long timeout) throws IOException {
        
        /* check ready keys */
        long startSelect = time.nanoseconds();
        //统计selector里面注册的kay数量
        int numReadyKeys = select(timeout);
        long endSelect = time.nanoseconds();
        this.sensors.selectTime.record(endSelect - startSelect, time.milliseconds());

        if (numReadyKeys > 0 || !immediatelyConnectedKeys.isEmpty() || dataInBuffers) {
            //取出key
            Set<SelectionKey> readyKeys = this.nioSelector.selectedKeys();

            // Poll from channels that have buffered data (but nothing more from the underlying socket)
            if (dataInBuffers) {
                keysWithBufferedRead.removeAll(readyKeys); //so no channel gets polled twice
                Set<SelectionKey> toPoll = keysWithBufferedRead;
                keysWithBufferedRead = new HashSet<>(); //poll() calls will repopulate if needed
                pollSelectionKeys(toPoll, false, endSelect);
            }

            // Poll from channels where the underlying socket has more data
            //遍历key 进行处理
            pollSelectionKeys(readyKeys, false, endSelect);
            // Clear all selected keys so that they are included in the ready count for the next select
            readyKeys.clear();

            pollSelectionKeys(immediatelyConnectedKeys, true, endSelect);
            immediatelyConnectedKeys.clear();
        } else {
            madeReadProgressLastPoll = true; //no work is also "progress"
        }

        long endIo = time.nanoseconds();
        this.sensors.ioTime.record(endIo - endSelect, time.milliseconds());

        // Close channels that were delayed and are now ready to be closed
        completeDelayedChannelClose(endIo);

        // we use the time at the end of select to ensure that we don't close any connections that
        // have just been processed in pollSelectionKeys
        maybeCloseOldestConnection(endSelect);

        // Add to completedReceives after closing expired connections to avoid removing
        // channels with completed receives until all staged receives are completed.
        addToCompletedReceives();
    }

selector.poll.pollSelectionKeys

这是最核心的方法，根据key的不同处理事件

OP_CONNECT：处理一些刚建立 tcp 连接的 channel

OP_WRITE：channel 发送消息

OP_READ: channel读取返回的数据

void pollSelectionKeys(Set<SelectionKey> selectionKeys,
                           boolean isImmediatelyConnected,
                           long currentTimeNanos) {
		//遍历key
        for (SelectionKey key : determineHandlingOrder(selectionKeys)) {
            //拿到channel
            KafkaChannel channel = channel(key);

            try {
				// key是OP_CONNECT就会执行，监听到连接事件
                if (isImmediatelyConnected || key.isConnectable()) {
                	//finishConnect
                    if (channel.finishConnect()) {
                    	//如果连接成功，channel放入conneted
                        this.connected.add(channel.id());
                        this.sensors.connectionCreated.record();
                        SocketChannel socketChannel = (SocketChannel) key.channel();
                        log.debug("Created socket with SO_RCVBUF = {}, SO_SNDBUF = {}, SO_TIMEOUT = {} to node {}",
                                socketChannel.socket().getReceiveBufferSize(),
                                socketChannel.socket().getSendBufferSize(),
                                socketChannel.socket().getSoTimeout(),
                                channel.id());
                    } else {
                        continue;
                    }
                }

                /* if channel is not ready finish prepare */
                if (channel.isConnected() && !channel.ready()) {
                    try {
                        channel.prepare();
                    } catch (AuthenticationException e) {
                        sensors.failedAuthentication.record();
                        throw e;
                    }
                    if (channel.ready())
                        sensors.successfulAuthentication.record();
                }

				//尝试读取
                attemptRead(key, channel);
                //处理发送请求的事件
                if (channel.ready() && key.isWritable()) {
                    Send send;
                    try {
                        send = channel.write();
                    } catch (Exception e) {
                        sendFailed = true;
                        throw e;
                    }
                    if (send != null) {
                        this.completedSends.add(send);
                        this.sensors.recordBytesSent(channel.id(), send.size());
                    }
                }
}

selector.poll.pollSelectionKeys.finishConnect

处理的是连接问题，这里面用了 transportLayer的finishConnect()，其实就是socketChannel的finishConnect()，把selector对应channel的连接事件去除，加入读事件，静待客户端数据返回。