《kafka producer学习笔记9》-NetworkClient

一 InFlightRequests

   上一篇我们整理了《kafka producer学习笔记8》-NIO，本篇其实对于producer而言，所剩不多了，主要就是networkclient了。有必要先补充下InFlightRequests，InFlightRequests队列的作用是缓存已经发出去但没有收到响应的ClientRequest。其底层是通过一个Map<String,Deque<ClientRequest>>对象实现，key是NodeId,value是发送到对应Node的ClientRequest对象集合。

/**
 * The set of requests which have been sent or are being sent but haven't yet received a response
 * 缓存了已经被发送或正在被发送但是均未接收到响应的客户端请求集合的一个封装
 */
final class InFlightRequests {
    //每个连接最大执行中请求数
    private final int maxInFlightRequestsPerConnection;
    //节点node至客户端请求双端队列Deque<ClientRequest>的映射集合,key为节点地址，Value为请求队列
    private final Map<String, Deque<NetworkClient.InFlightRequest>> requests = new HashMap<>();
    /** Thread safe total number of in flight requests. */
    private final AtomicInteger inFlightRequestCount = new AtomicInteger(0);
...
}

当有新请求需要处理时，会在队首入列，而实际被处理的请求，则是从队尾出列，保证入列早的请求先得到处理。由于sendmore的条件限制，使用队列虽然可以存储多个请求，但是新的请求能加进来的条件是上一个请求必须已经发送成功。

先看下发送条件限制,NetworkClient调用这个方法是用于判断是否可以向指定Node发送请求的条件之一

   InFlightRequests.java
 /**
     * Can we send more requests to this node?
     *
     * @param node Node in question
     * @return true iff we have no requests still being sent to the given node
     * 判断是否该连接能发送请求
     */
    public boolean canSendMore(String node) {
        Deque<NetworkClient.InFlightRequest> queue = requests.get(node);
        //判断条件：队列为空
        return queue == null || queue.isEmpty() ||
        		//或者队列头请求已经发送完成且队列中没有堆积过多请求
               (queue.peekFirst().send.completed() && queue.size() < this.maxInFlightRequestsPerConnection);
    }

        一开始是队列为空的判断条件为空就能发送，接着queue.peekFirst().request().completed()：这个条件为true表示当前队头的请求已经发送完成，如果队头的请求迟迟发送不出去，可能是网络的原因，则不能继续向此Node发送请求。而且，队头的消息与对应KafkaChannel.send字段指向的事同一个消息，为了避免未发送的消息被覆盖，也不能让KafkaChannel.send字段指向新请求。queue.size() < this.maxInFlightRequestsPerConnection：为了判断InFlightRequests队列中是否堆积过多请求。如果Node已经堆积了很多未响应的请求，说明这个节点的负载或网络连接有问题，继续发送请求，则可能会超时。
 

  再看下入队，入队是通过add()方法来完成的，代码如下：

InFlightRequests.java
/**
     * Add the given request to the queue for the connection it was directed to
     * 表示已经发送，或正在发送。并且还没有收到响应的（客户端）请求。请求首先加入到（目标节点对应的）队列头部
     * 注意：由于sendmore()的限制，但是新的请求能加进来的条件是上一个请求必须已经发送成功！（这就避免了因为网络阻塞，请求一直堆积在某个节点上。）
     */
    public void add(NetworkClient.InFlightRequest request) {
    	//这个请求要发送到哪个Broker节点上
        String destination = request.destination;
        // 从requests集合中根据给定请求的目的地node获取Deque<ClientRequest>双端队列reqs
        Deque<NetworkClient.InFlightRequest> reqs = this.requests.get(destination);
        if (reqs == null) {// 如果双端队列reqs为nul
        	// 构造一个双端队列ArrayDeque类型的reqs
            reqs = new ArrayDeque<>();
            // 将请求目的地node至reqs的对应关系添加到requests集合
            this.requests.put(destination, reqs);
        }
        // reqs队列首部添加请求request，使用的是addFirst()方法加入队首
        reqs.addFirst(request);
        inFlightRequestCount.incrementAndGet();//计数增加
    }

再看下出队，是有NetworkClient.handleCompletedReceives实现响应完成时调用，从inFlightRequests队列删除。

InFlightRequests.java 
/**
     * Get the oldest request (the one that will be completed next) for the given node
     * 获取给定节点node的时间最久执行中请求，作为接下来要完成的请求
     */
    public NetworkClient.InFlightRequest completeNext(String node) {
    	// 根据给定节点node获取客户端请求双端队列reqs，并从poll出队尾元素
        NetworkClient.InFlightRequest inFlightRequest = requestQueue(node).pollLast();
        inFlightRequestCount.decrementAndGet();//计数器-1
        return inFlightRequest;
    }

对比下，入队add时是通过addFirst()方法添加到队首的，所以队尾的元素是时间最久的，也是应该先处理的，故出队应该用pollLast()，将存储时间最久的元素移出进行处理。另外注意一点：peekFirst与pollFirst区别。

    /**
     * Get the last request we sent to the given node (but don't remove it from the queue)
     * @param node The node id
     *（Deque是个双端队列，头尾操作方便）最后发送的请求
     */
    public NetworkClient.InFlightRequest lastSent(String node) {
        return requestQueue(node).peekFirst();
    }

    /**
     * Complete the last request that was sent to a particular node.
     *  取出该连接，最新的请求
     * @param node The node the request was sent to
     * @return The request
     */
    public NetworkClient.InFlightRequest completeLastSent(String node) {
        NetworkClient.InFlightRequest inFlightRequest = requestQueue(node).pollFirst();
        inFlightRequestCount.decrementAndGet();
        return inFlightRequest;
    }

 InFlightRequest表示正在发送的请求，它存储着请求发送前的所有的信息。另外，它还支持生成响应ClientResponse。当正常收到响应时，completed 方法会根据响应内容生成ClientResponse。当连接突然断开，disconnected方法会生成ClientResponse。

NetworkClient.InFlightRequest.java  
static class InFlightRequest {
    	 // 请求头
        final RequestHeader header;
        //表示这个请求要发送到哪个Broker节点上
        final String destination;
        // 回调函数
        final RequestCompletionHandler callback;
        // 是否需要服务端返回响应
        final boolean expectResponse;
        // 请求体
        final AbstractRequest request;
        // 表示发送前是否需要验证连接状态
        final boolean isInternalRequest; // used to flag requests which are initiated internally by NetworkClient
        // 请求的序列化数据
        final Send send;
        // 发送时间
        final long sendTimeMs;
        // 请求的创建时间，这个是ClientRequest的创建时间
        final long createdTimeMs;
        //请求超时时间
        final long requestTimeoutMs;
       ....

  /**
         * 收到响应，回调的时候据响应内容生成ClientResponse
         */
        public ClientResponse completed(AbstractResponse response, long timeMs) {
            return new ClientResponse(header, callback, destination, createdTimeMs, timeMs,
                    false, null, null, response);
        }
        /**
         * 当连接突然断开，disconnected方法会生成ClientResponse。
         * @param timeMs
         * @param authenticationException
         * @return
         */
        public ClientResponse disconnected(long timeMs, AuthenticationException authenticationException) {
            return new ClientResponse(header, callback, destination, createdTimeMs, timeMs,
                    true, null, authenticationException, null);
        }
}

二 NetworkClient

  NetworkClient依赖的组件都介绍完了，现在看下NetworkClient的实现。NetworkClient是个通用的网络客户端的实现，Kafka的所有消息，都是通过NetworkClient发送消息。无论是Kafka的生产者，还是Kakfa的消费者，都会包含NetworkClient。客户端和服务端的交互两种形式：发送请求和读取响应。

 

2.1 属性与方法 

public class NetworkClient implements KafkaClient {

    private final Logger log;

    /* the selector used to perform network i/o 
     * Kafka的Selector ，用来进行网络io操作
     * */
    private final Selectable selector;

    private final MetadataUpdater metadataUpdater;

    private final Random randOffset;

    /* the state of each node's connection 
     * 保存与每个节点的连接状态
     * */
    private final ClusterConnectionStates connectionStates;

    /* the set of requests currently being sent or awaiting a response */
    //请求队列，保存正在发送但还没有收到响应的请求
    private final InFlightRequests inFlightRequests;

    /* the socket send buffer size in bytes */
    private final int socketSendBuffer;

    /* the socket receive size buffer in bytes */
    private final int socketReceiveBuffer;

    /* the client id used to identify this client in requests to the server */
    private final String clientId;

    /* the current correlation id to use when sending requests to servers */
    private int correlation;

    /* default timeout for individual requests to await acknowledgement from servers */
    private final int defaultRequestTimeoutMs;

    /* time in ms to wait before retrying to create connection to a server */
    private final long reconnectBackoffMs;

    private final ClientDnsLookup clientDnsLookup;

    private final Time time;

    /**
     * True if we should send an ApiVersionRequest when first connecting to a broker.
     * 是否需要与服务器的版本协调，默认都为true.第一次与broker连接需要
     */
    private final boolean discoverBrokerVersions;

    private final ApiVersions apiVersions;
    /**
     * 存储着要发送的版本请求，Key为主机地址，Value为构建请求的Builder
     */
    private final Map<String, ApiVersionsRequest.Builder> nodesNeedingApiVersionsFetch = new HashMap<>();
...
}

NetworkClient 主要方法有

1. 调用NetworkClient的ready方法，连接服务端
2. 调用NetworkClient的poll方法，处理连接
3. 调用NetworkClient的newClientRequest方法，创建请求ClientRequest
4. 然后调用NetworkClient的send方法，发送请求
5. 最后调用NetworkClient的poll方法，处理响应

 networkclient流程：

2.2 ready()

NetworkClient发送请求之前，都需要先和服务端创建连接。如果没有ready，就会从readNodes里面移除，接下来就不会往这个Node发送消息。下面看看ready()

NetworkClient.java   
 public boolean ready(Node node, long now) {
        if (node.isEmpty())
            throw new IllegalArgumentException("Cannot connect to empty node " + node);
        //是否准备好发送请求
        if (isReady(node, now))
            return true;
        //isReady()为false，会先初始化连接initiateConnect
        if (connectionStates.canConnect(node.idString(), now))
            // if we are interested in sending to a node and we don't have a connection to it, initiate one
            initiateConnect(node, now);

        return false;
    }
/**
     * Check if the node with the given id is ready to send more requests.
     *
     * @param node The node
     * @param now The current time in ms
     * @return true if the node is ready
     */
    @Override
    public boolean isReady(Node node, long now) {
        // if we need to update our metadata now declare all requests unready to make metadata requests first
        // priority  当发现正在更新元数据时，会禁止发送请求&& 当连接没有创建完毕或者当前发送的请求过多时，也会禁止发送请求
        return !metadataUpdater.isUpdateDue(now) && canSendRequest(node.idString(), now);
    }

    /**
     * Are we connected and ready and able to send more requests to the given connection?
     * 检测连接状态，检测发送请求是否过多
     * @param node The node
     * @param now the current timestamp
     */
    private boolean canSendRequest(String node, long now) {
        return connectionStates.isReady(node, now) && selector.isChannelReady(node) &&
            inFlightRequests.canSendMore(node);
    }

      /**
     * Initiate a connection to the given node
     * 创建连接到指定节点
     */
    private void initiateConnect(Node node, long now) {
        String nodeConnectionId = node.idString();
        try {
        	 // 更新连接状态为正在连接
            this.connectionStates.connecting(nodeConnectionId, now, node.host(), clientDnsLookup);
            InetAddress address = this.connectionStates.currentAddress(nodeConnectionId);
            log.debug("Initiating connection to node {} using address {}", node, address);
            // 调用selector异步连接
            selector.connect(nodeConnectionId,
                    new InetSocketAddress(address, node.port()),
                    this.socketSendBuffer,
                    this.socketReceiveBuffer);
        } catch (IOException e) {
            /* attempt failed, we'll try again after the backoff */
            connectionStates.disconnected(nodeConnectionId, now);
            /* maybe the problem is our metadata, update it */
            metadataUpdater.requestUpdate();
            log.warn("Error connecting to node {}", node, e);
        }
    }

   首先通过NetworkClient.isReady()方法检测是否能向一个Node发送请求，需要符合以下三个条件，表示Node已经准备好：

• Metadata并未处于正在更新或需要更新的状态。
• 已经成功建立连接并连接正常connectionStates.isConnected(node)
• InFlightRequests.canSendMore()返回true。
  如果NetworkClient.isReady()返回false,且满足下面两个条件，就会调用initiateConnect()方法重新连接。

   ClusterConnectionStates.java
 /**
     * Return true iff we can currently initiate a new connection. This will be the case if we are not
     * connected and haven't been connected for at least the minimum reconnection backoff period.
     * @param id the connection id to check
     * @param now the current time in ms
     * @return true if we can initiate a new connection
     */
    public boolean canConnect(String id, long now) {
        NodeConnectionState state = nodeState.get(id);
        if (state == null)
            return true;
        else
            return state.state.isDisconnected() &&
                   now - state.lastConnectAttemptMs >= state.reconnectBackoffMs;
    }

• 连接不能是CONNECTEDING状态，必须是DISCONNECTED。
• 为了避免网络拥塞，重连不能太频繁，两次重试之间的时间差必须大于重试的退避时间，由reconnectBackoffMs指定。

         之前在看Selector部分的时候，一旦有读操作，就要读取一个完整的NetworkReceive。如果是写，可以分多次写。即读操作会在一次SelectionKey循环读取一个完整的接收动作，而写操作会在多次SelectionKey中完成一个完整的发送动作。这里ready相关，一个channel里面的Send对象要是只发送了部分，下次就不会处于ready状态了。

2.2 send()

将请求设置到KafkaChannel.send字段，同时将请求添加到InFlightRequests队列中等待响应。

先看下请求,NetworkClient使用ClientRequest类表示请求

/**
 * A request being sent to the server. This holds both the network send as well as the client-level metadata.
 * ClientRequest是客户端的请求，这个请求会被发送到服务器，所以封装了requestBuilder。
 */
public final class ClientRequest {
	// 节点地址
    private final String destination;
    //ClientRequest中通过requetBuilder给不同类型的请求设置不同的请求内容
    private final AbstractRequest.Builder<?> requestBuilder;
    // 请求头的correlation id
    private final int correlationId;
    // 请求头的client id
    private final String clientId;
    // 创建时间
    private final long createdTimeMs;
    //是否需要响应
    private final boolean expectResponse;
    //请求的超时时间
    private final int requestTimeoutMs;
    // 回调函数，用于处理响应
    private final RequestCompletionHandler callback;
   ...
}

   NetworkClient.java
 /**
     * Queue up the given request for sending. Requests can only be sent out to ready nodes.
     * @param request The request
     * @param now The current timestamp
     *  发送请求,这个方法：producer跟consumer 都会调用，ClientRequest 表示客户端的请求，它是嵌在producer、consumer里面的。
     */
    @Override
    public void send(ClientRequest request, long now) {
        doSend(request, false, now);
    }
     private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long now) {
        String nodeId = clientRequest.destination();
        if (!isInternalRequest) {
            // If this request came from outside the NetworkClient, validate
            // that we can send data.  If the request is internal, we trust
            // that internal code has done this validation.  Validation
            // will be slightly different for some internal requests (for
            // example, ApiVersionsRequests can be sent prior to being in
            // READY state.)
            if (!canSendRequest(nodeId, now))//检测是否能够向指定Node发送请求
                throw new IllegalStateException("Attempt to send a request to node " + nodeId + " which is not ready.");
        }
        AbstractRequest.Builder<?> builder = clientRequest.requestBuilder();
        try {
        	//检测版本
            NodeApiVersions versionInfo = apiVersions.get(nodeId);
            short version;
            // Note: if versionInfo is null, we have no server version information. This would be
            // the case when sending the initial ApiVersionRequest which fetches the version
            // information itself.  It is also the case when discoverBrokerVersions is set to false.
            if (versionInfo == null) {
                version = builder.latestAllowedVersion();
                if (discoverBrokerVersions && log.isTraceEnabled())
                    log.trace("No version information found when sending {} with correlation id {} to node {}. " +
                            "Assuming version {}.", clientRequest.apiKey(), clientRequest.correlationId(), nodeId, version);
            } else {
                version = versionInfo.latestUsableVersion(clientRequest.apiKey(), builder.oldestAllowedVersion(),
                        builder.latestAllowedVersion());
            }
            // The call to build may also throw UnsupportedVersionException, if there are essential
            // fields that cannot be represented in the chosen version.
            // builder.build()是ProduceRequest.Builder ，结果是ProduceRequest
            doSend(clientRequest, isInternalRequest, now, builder.build(version));
        } catch (UnsupportedVersionException unsupportedVersionException) {
            // If the version is not supported, skip sending the request over the wire.
            // Instead, simply add it to the local queue of aborted requests.
        	// 版本不支持，不发送，生成clientResponse，添加到abortedSends集合里
            log.debug("Version mismatch when attempting to send {} with correlation id {} to {}", builder,
                    clientRequest.correlationId(), clientRequest.destination(), unsupportedVersionException);
            ClientResponse clientResponse = new ClientResponse(clientRequest.makeHeader(builder.latestAllowedVersion()),
                    clientRequest.callback(), clientRequest.destination(), now, now,
                    false, unsupportedVersionException, null, null);
            abortedSends.add(clientResponse);
        }
    }
     private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long now, AbstractRequest request) {
    	//目标节点
        String destination = clientRequest.destination();
        // 生成请求头
        RequestHeader header = clientRequest.makeHeader(request.version());
        if (log.isDebugEnabled()) {
            int latestClientVersion = clientRequest.apiKey().latestVersion();
            if (header.apiVersion() == latestClientVersion) {
                log.trace("Sending {} {} with correlation id {} to node {}", clientRequest.apiKey(), request,
                        clientRequest.correlationId(), destination);
            } else {
                log.debug("Using older server API v{} to send {} {} with correlation id {} to node {}",
                        header.apiVersion(), clientRequest.apiKey(), request, clientRequest.correlationId(), destination);
            }
        }
        //toSend()是一个接口，返回的是NetworkSend（继承了ByteBufferSend）
        // 结合请求头和请求体，序列化数据，保存到NetworkSend
        Send send = request.toSend(destination, header);
        //生成InFlightRequest实例，它保存了正在发送中的请求
        InFlightRequest inFlightRequest = new InFlightRequest(
                clientRequest,
                header,
                isInternalRequest,
                request,
                send,
                now);
        //先加入到inFlightRequests队列中
        this.inFlightRequests.add(inFlightRequest);
        //调用Selector异步发送数据（将send和对应kafkaChannel绑定起来，并开启该kafkaChannel底层socket的写事件）
        selector.send(send);
    }

2.3 poll()

selector.poll函数，会把处理结果，放到一堆的状态变量里面（输出结果集）.就是之前pollSelectionKeys各种读写。NetworkClient这里使用handle*()方法对结果进行处理

 NetworkClient.java  
 /**
     * Do actual reads and writes to sockets.
     *
     * @param timeout The maximum amount of time to wait (in ms) for responses if there are none immediately,
     *                must be non-negative. The actual timeout will be the minimum of timeout, request timeout and
     *                metadata timeout
     * @param now The current time in milliseconds
     * @return The list of responses received
     * 关于 socket 的一些实际的读写操作
     * 涉及： Metadata 操作、Selector 进行相应的 IO 操作；处理 Server 端的 response 
     */
    @Override
    public List<ClientResponse> poll(long timeout, long now) {
        if (!abortedSends.isEmpty()) {
            // If there are aborted sends because of unsupported version exceptions or disconnects,
            // handle them immediately without waiting for Selector#poll.
            List<ClientResponse> responses = new ArrayList<>();
            handleAbortedSends(responses);
            completeResponses(responses);
            return responses;
        }
        // 1判断是否需要更新 meta（这里参照之前的metadata更新）
        long metadataTimeout = metadataUpdater.maybeUpdate(now);
        try {//2调用selector的poll方法， 真正读写发生的地方. 如果客户端请求被完整地处理过了, 会加入到completeSends或complteReceives中
            this.selector.poll(Utils.min(timeout, metadataTimeout, defaultRequestTimeoutMs));
        } catch (IOException e) {
            log.error("Unexpected error during I/O", e);
        }

        // process completed actions 处理已经完成的动作
        long updatedNow = this.time.milliseconds();
        //响应列表：真正的读写操作, 会生成responses
        List<ClientResponse> responses = new ArrayList<>();
        // 3处理各种情况，生成响应，添加到列表中
        //完成发送的handler（不需要 response 的 request,如 send）
        handleCompletedSends(responses, updatedNow);
        //完成接收的handler（如 Metadata 请求）
        handleCompletedReceives(responses, updatedNow);
        //断开连接的handler
        handleDisconnections(responses, updatedNow);
        //处理连接的handler
        handleConnections();
        //处理版本协调请求（获取api版本号）handler
        handleInitiateApiVersionRequests(updatedNow);
        //超时请求的handler
        handleTimedOutRequests(responses, updatedNow);
        //invoke callback 触发回调函数的调用
        completeResponses(responses);

        return responses;
    }

各种handle.先看下处理连接的,poll 方法会调用 handleConnections处理连接,并且会创建版本请求。。

NetworkClient.java
    /**
     * Record any newly completed connections
     * 处理连接
     */
    private void handleConnections() {
    	 // 遍历刚创建完成的连接
        for (String node : this.selector.connected()) {
            // We are now connected.  Node that we might not still be able to send requests. For instance,
            // if SSL is enabled, the SSL handshake happens after the connection is established.
            // Therefore, it is still necessary to check isChannelReady before attempting to send on this
            // connection.
            if (discoverBrokerVersions) {
            	 // 更新连接的状态为版本协调状态
                this.connectionStates.checkingApiVersions(node);
                 // 将请求保存到nodesNeedingApiVersionsFetch集合里
                nodesNeedingApiVersionsFetch.put(node, new ApiVersionsRequest.Builder());
                log.debug("Completed connection to node {}. Fetching API versions.", node);
            } else {
                this.connectionStates.ready(node);
                log.debug("Completed connection to node {}. Ready.", node);
            }
        }
    }

创建完版本请求，接下来看看是如何发送请求和处理响应的。poll方法会调用handleInitiateApiVersionRequests发送版本协调请求，然后调用handleApiVersionsResponse负责处理响应。

NetworkClient.java 
// 发送版本协调请求
    private void handleInitiateApiVersionRequests(long now) {
    	// 遍历请求集合nodesNeedingApiVersionsFetch(在handleConnections加入的)
        Iterator<Map.Entry<String, ApiVersionsRequest.Builder>> iter = nodesNeedingApiVersionsFetch.entrySet().iterator();
        while (iter.hasNext()) {
            Map.Entry<String, ApiVersionsRequest.Builder> entry = iter.next();
            String node = entry.getKey();
            // 判断是否允许发送请求
            if (selector.isChannelReady(node) && inFlightRequests.canSendMore(node)) {
                log.debug("Initiating API versions fetch from node {}.", node);
                ApiVersionsRequest.Builder apiVersionRequestBuilder = entry.getValue();
                // 调用newClientRequest生成请求
                ClientRequest clientRequest = newClientRequest(node, apiVersionRequestBuilder, now, true);
                // 发送请求
                doSend(clientRequest, true, now);
                iter.remove();
            }
        }
    }
    private void handleCompletedReceives(List<ClientResponse> responses, long now) {
...
 //判断是否为MetadataResponse
            if (req.isInternalRequest && body instanceof MetadataResponse)
                metadataUpdater.handleCompletedMetadataResponse(req.header, now, (MetadataResponse) body);
            else if (req.isInternalRequest && body instanceof ApiVersionsResponse)
            	//处理办理请求响应ApiVersionsResponse
                handleApiVersionsResponse(responses, req, now, (ApiVersionsResponse) body);
            else //其他响应：生成ClientResponse添加到列表中
                responses.add(req.completed(body, now));
        }
    }
    //处理版本请求响应
    private void handleApiVersionsResponse(List<ClientResponse> responses,
                                           InFlightRequest req, long now, ApiVersionsResponse apiVersionsResponse) {
    	//目标节点
        final String node = req.destination;
        // 判断响应和版本是否协调
        if (apiVersionsResponse.error() != Errors.NONE) {
        	//异常处理
            if (req.request.version() == 0 || apiVersionsResponse.error() != Errors.UNSUPPORTED_VERSION) {
                log.warn("Received error {} from node {} when making an ApiVersionsRequest with correlation id {}. Disconnecting.",
                        apiVersionsResponse.error(), node, req.header.correlationId());
                this.selector.close(node);
                processDisconnection(responses, node, now, ChannelState.LOCAL_CLOSE);
            } else {
                nodesNeedingApiVersionsFetch.put(node, new ApiVersionsRequest.Builder((short) 0));
            }
            return;
        }
        NodeApiVersions nodeVersionInfo = new NodeApiVersions(apiVersionsResponse.apiVersions());
        //更新版本
        apiVersions.update(node, nodeVersionInfo);
        //连接状态更新为ready，可以正常发送请求了。
        this.connectionStates.ready(node);
        log.debug("Recorded API versions for node {}: {}", node, nodeVersionInfo);
    }

上面看了send是请求发送，那么对应的处理响应的过程如下：

NetworkClient在发送完set/get请求后，就会调用handleCompletedSends，表示请求已经发送到服务端了。至于请求在服务端被处理，什么时候完成完全取决于服务端，当get请求收到响应时，才调用handleCompletedReceives。

handleCompletedSends

NetworkClient.java  
  /**
     * Handle any completed request send. In particular if no response is expected consider the request complete.
     *
     * @param responses The list of responses to update
     * @param now The current time
     * 客户端发送请求后，handleCompletedSends中对于有响应的请求，并不会将ClientRequest从inFlightRequests中移除。（因为没收到响应）
     * 除非是客户端请求不需要响应，则这时候是可以将ClientRequest从中删除，添加时放到头部，删除时也是从头部删除。
     */
    private void handleCompletedSends(List<ClientResponse> responses, long now) {
        // if no response is expected then when the send is completed, return it
        for (Send send : this.selector.completedSends()) {
        	 // 这里获取目标节点的队列中第一个请求, 但并没有从队列中删除, 取出之后判断这个请求是否期望得到响应
            InFlightRequest request = this.inFlightRequests.lastSent(send.destination());
            // 如果不需要响应,当Send请求完成时,就直接返回.还是有request.completed生成的ClientResponse对象
            if (!request.expectResponse) {
                this.inFlightRequests.completeLastSent(send.destination());
                // 调用completed方法生成ClientResponse,completed（）第一个参数为null,表示没有响应内容
                responses.add(request.completed(null, now));
            }
            // 如果客户端请求需要有响应, 那么它的响应是在下面的handleCompletedReceives中设置的.  
        }
    }

怎么能保证从Selector返回的请求，是对应到队列中最新的请求?

     InFlightRequests 中存储的是已发送但没收到响应的请求，而 completedSends 保存的是最近一次 poll() 方法中发送成功的请求（发送成功并没有收到响应）. 每个请求发送，都要等待前面的请求发送完成，这样就能保证同一时间只有一个请求正在发送 。这样就能对应上。

当请求收到响应后，会触发handleCompletedReceives函数

handleCompletedReceives

NetworkClient.java 
/**
     * Handle any completed receives and update the response list with the responses received.
     *
     * @param responses The list of responses to update
     * @param now The current time
     */
    private void handleCompletedReceives(List<ClientResponse> responses, long now) {
    	//遍历响应，通过Selector返回未处理的响应
        for (NetworkReceive receive : this.selector.completedReceives()) {
            String source = receive.source();
            //从缓存队列获取已发送请求并删除（这里会从inFlightRequests删除，因为inFlightRequests存的是未收到请求的ClientRequest，现在这个请求已经有响应了，就不需要再其中保存了。）
            InFlightRequest req = inFlightRequests.completeNext(source);
            //解析响应,并且验证响应头，生成Struct实例
            Struct responseStruct = parseStructMaybeUpdateThrottleTimeMetrics(receive.payload(), req.header,
                throttleTimeSensor, now);
            if (log.isTraceEnabled()) {
                log.trace("Completed receive from node {} for {} with correlation id {}, received {}", req.destination,
                    req.header.apiKey(), req.header.correlationId(), responseStruct);
            }
            // If the received response includes a throttle delay, throttle the connection.
            // 生成响应体
            AbstractResponse body = AbstractResponse.parseResponse(req.header.apiKey(), responseStruct);
            maybeThrottle(body, req.header.apiVersion(), req.destination, now);
            //判断是否为MetadataResponse
            if (req.isInternalRequest && body instanceof MetadataResponse)
                metadataUpdater.handleCompletedMetadataResponse(req.header, now, (MetadataResponse) body);
            else if (req.isInternalRequest && body instanceof ApiVersionsResponse)
            	//处理办理请求响应ApiVersionsResponse
                handleApiVersionsResponse(responses, req, now, (ApiVersionsResponse) body);
            else //其他响应：生成ClientResponse添加到列表中
                responses.add(req.completed(body, now));
        }

    private static Struct parseStructMaybeUpdateThrottleTimeMetrics(ByteBuffer responseBuffer, RequestHeader requestHeader,
                                                                    Sensor throttleTimeSensor, long now) {
    	// 解析响应头
        ResponseHeader responseHeader = ResponseHeader.parse(responseBuffer);
        // Always expect the response version id to be the same as the request version id
        // 解析响应体
        Struct responseBody = requestHeader.apiKey().parseResponse(requestHeader.apiVersion(), responseBuffer);
        // 验证请求头与响应头的 correlation id 必须相等
        correlate(requestHeader, responseHeader);
        if (throttleTimeSensor != null && responseBody.hasField(CommonFields.THROTTLE_TIME_MS))
            throttleTimeSensor.record(responseBody.get(CommonFields.THROTTLE_TIME_MS), now);
        return responseBody;
    }

收到响应是在handleCompletedReceives，这时候才可以调用completeNext删除source对应的ClientRequest，因为我们知道inFlightRequests存的是未收到请求的ClientRequest，现在这个请求已经有响应了，就不需要再其中保存了。那么inFlightRequests久起到了可以防止请求堆积的作用。对比一下两种方式：

执行回调函数completeResponses

     NetworkClient的 poll 方法，会将已完成的请求，生成ClientReponse收集起来，然后逐一执行它的回调函数。如果是异常响应则要求重发，如果是正常响应则调用每个消息自定义的Callback。在creatteProduceRequests()方法中提到过，这里调用的Callback回调对象，也就是RequestCompletionHandler对象，其onComplete()方法最终调用Sender.handleProduceResponse()方法。

NetworkClient.java  
  private void completeResponses(List<ClientResponse> responses) {
        for (ClientResponse response : responses) {
            try {//调用response的完成
                response.onComplete();
            } catch (Exception e) {
                log.error("Uncaught error in request completion:", e);
            }
        }
    }

public class ClientResponse {
	
	//请求头
    private final RequestHeader requestHeader;
    // 回调函数，由ClientRequest指定
    private final RequestCompletionHandler callback;
    //目标节点地址
    private final String destination;
    //接受时间
    private final long receivedTimeMs;
    private final long latencyMs;    
    private final boolean disconnected;
    private final UnsupportedVersionException versionMismatch;
    private final AuthenticationException authenticationException;
    // 响应体
    private final AbstractResponse responseBody;
...
  // 响应完成的回调函数
    public void onComplete() {
        if (callback != null)  //调用RequestCompletionHandler回调
            callback.onComplete(this);
    }

}

 沿着sender.handleProduceResponse()参照之前的文章，不在重复了。