kafka消费者（一）：ConsumerNetworkClient

ConsumerNetworkClient在NetworkClient上作了封装，核心字段如下：

public class ConsumerNetworkClient implements Closeable {
    //NetworkClient对象
    private final KafkaClient client;
    //调用KafkaConsumer对象的消费者线程之外的其他线程设置，表示要中断KafkaConsumer线程
    private final AtomicBoolean wakeup = new AtomicBoolean(false);
    //定时任务队列，非线程安全、无界的、优先级队列。这个定时任务是心跳任务。
    private final DelayedTaskQueue delayedTasks = new DelayedTaskQueue();
    //缓冲队列，key是Node节点，value是发往这个Node的ClientRequest集合。
    private final Map<Node, List<ClientRequest>> unsent = new HashMap<>();
    //管理Kafka集群的元数据
    private final Metadata metadata;
    private final Time time;
    private final long retryBackoffMs;
    //unsent中缓存的超时时长
    private final long unsentExpiryMs;

    //KafkaConsumer是否在执行不可中断的方法，每进入一个不可中断的方法则加一，退出则减一。只会被KafkaConsumer线程修改。
    private int wakeupDisabledCount = 0;
}

ConsumerNetworkClient.poll()是最核心的方法，poll有多个重载，最终会调用private void poll(long timeout, long now, boolean executeDelayedTasks)。
其中timeout表示poll方法的最长阻塞时间，now表示当前时间戳，executeDelayedTasks表示是否执行delayTasks中的定时任务。下面介绍其流程：
(一) trySend()方法循环处理unsent中缓存的请求。对于每个Node节点，遍历对应的ClientRequest列表，每次循环都调用ready()方法检测节点之间的连接，以及发送的条件。若符合发送要求，就调用NetworkClient.send()方法把请求放到InFlightRequest中等待响应，也放入KafkaChannel的send字段中等待发送。
 

private boolean trySend(long now) {
    // send any requests that can be sent now
    boolean requestsSent = false;
    //遍历unsend集合
    for (Map.Entry<Node, List<ClientRequest>> requestEntry: unsent.entrySet()) {
        Node node = requestEntry.getKey();
        Iterator<ClientRequest> iterator = requestEntry.getValue().iterator();
        while (iterator.hasNext()) {
            ClientRequest request = iterator.next();
            //调用NetworkClient.ready()查看是否可以发送请求。
            if (client.ready(node, now)) {
                //调用send()方法等待发送请求
                client.send(request, now);
                //从unsent中删除此请求
                iterator.remove();
                requestsSent = true;
            }
        }
    }
    return requestsSent;
}

(二) 计算超时时间，根据timeout和delayedTasks队列中最近要执行的定时任务决定。
(三) 调用poll()方法，把send字段发送出去。可能会更新Metadata使用一系列handle*()方法处理请求响应、连接断开、超时等情况，调用请求的回调函数。
(四) 调用maybeTriggerWakeup()方法检测wakeup和wakeupDisabledCount，查看是否有其他线程中断。
 

private void maybeTriggerWakeup() {
    //通过wakeupDisabledCount检测是否在执行不可中断的方法，通过wakeup检测是否有中断请求
    if (wakeupDisabledCount == 0 && wakeup.get()) {
        //设置中断标志
        wakeup.set(false);
        throw new WakeupException();
    }
}

(五) 调用checkDisconnect()方法检测连接状态。当检测到连接断开的node时，把他在unsent集合中对应的全部ClientRequest的对象清除掉。然后调用这些clientRequest的回调函数。
 

private void checkDisconnects(long now) {
    // any disconnects affecting requests that have already been transmitted will be handled
    // by NetworkClient, so we just need to check whether connections for any of the unsent
    // requests have been disconnected; if they have, then we complete the corresponding future
    // and set the disconnect flag in the ClientResponse
    Iterator<Map.Entry<Node, List<ClientRequest>>> iterator = unsent.entrySet().iterator();
    while (iterator.hasNext()) {//遍历unsent集合的每个Node
        Map.Entry<Node, List<ClientRequest>> requestEntry = iterator.next();
        Node node = requestEntry.getKey();
        if (client.connectionFailed(node)) {//检测消费者与每个Node之间的连接状态
            // Remove entry before invoking request callback to avoid callbacks handling
            // coordinator failures traversing the unsent list again.
            //从unsent集合中删除这个Node的所有ClientRequest
            iterator.remove();
            for (ClientRequest request : requestEntry.getValue()) {
                RequestFutureCompletionHandler handler =
                        (RequestFutureCompletionHandler) request.callback();
                // 调用ClientRequest函数。
                handler.onComplete(new ClientResponse(request, now, true, null));
            }
        }
    }
}

(六) 根据executeDelayTasks参数决定是否处理delayedTasks队列中超时的定时任务，如果需要就调用delayedTasks.poll()方法
(七) 再次调用trySend()方法，在步骤三中调用了poll方法，可能已经把send字段的请求发送出去了，也可能新建立了某些Node的网络连接，所以再次调用trySend()方法。
(八) 调用failExpiredRequests处理unsent中的超时请求。
 

private void failExpiredRequests(long now) {
    // clear all expired unsent requests and fail their corresponding futures
    Iterator<Map.Entry<Node, List<ClientRequest>>> iterator = unsent.entrySet().iterator();
    while (iterator.hasNext()) {//遍历unsent集合，
        Map.Entry<Node, List<ClientRequest>> requestEntry = iterator.next();
        Iterator<ClientRequest> requestIterator = requestEntry.getValue().iterator();
        while (requestIterator.hasNext()) {
            ClientRequest request = requestIterator.next();
            if (request.createdTimeMs() < now - unsentExpiryMs) {//检查是否超时
                RequestFutureCompletionHandler handler =
                        (RequestFutureCompletionHandler) request.callback();
                //调用回调函数
                handler.raise(new TimeoutException("Failed to send request after " + unsentExpiryMs + " ms."));
                //删除ClientRequest
                requestIterator.remove();
            } else
                break;
        }
        //队列已经清空，就从unsent中删除
        if (requestEntry.getValue().isEmpty())
            iterator.remove();
    }
}

现在我们再看回poll()方法：

private void poll(long timeout, long now, boolean executeDelayedTasks) {
    // send all the requests we can send now
    trySend(now);

    // ensure we don't poll any longer than the deadline for
    // the next scheduled task
    timeout = Math.min(timeout, delayedTasks.nextTimeout(now));
    clientPoll(timeout, now);
    now = time.milliseconds();

    // handle any disconnects by failing the active requests. note that disconnects must
    // be checked immediately following poll since any subsequent call to client.ready()
    // will reset the disconnect status
    checkDisconnects(now);

    // execute scheduled tasks
    if (executeDelayedTasks)
        delayedTasks.poll(now);

    // try again to send requests since buffer space may have been
    // cleared or a connect finished in the poll
    trySend(now);

    // fail requests that couldn't be sent if they have expired
    failExpiredRequests(now);
}

需要重点关注的是KafkaConsumer中使用的回调对象--RequestFutureCompletionHandler，只有omComplet()方法，如下：
 

public static class RequestFutureCompletionHandler
        extends RequestFuture<ClientResponse>
        implements RequestCompletionHandler {

    @Override
    public void onComplete(ClientResponse response) {
        //因连接故障而产生的ClientResponse对象
        if (response.wasDisconnected()) {
            ClientRequest request = response.request();
            RequestSend send = request.request();
            ApiKeys api = ApiKeys.forId(send.header().apiKey());
            int correlation = send.header().correlationId();
            log.debug("Cancelled {} request {} with correlation id {} due to node {} being disconnected",
                    api, request, correlation, send.destination());
            //调用继承自父类的RequestFutrue的raise()方法。
            raise(DisconnectException.INSTANCE);
        } else {
            //调用继承自父类的RequestFutrue的complete()方法。
            complete(response);
        }
    }
}

ConsumerNetworkClient还提供了几个常用的功能。
awaitMetadataUpdate：循环调用poll方法，直到Metadata版本号+1
awaitPendingRequests：等待unsent和InFightRequest中请求全部完成
put：向unsent中添加请求。
schedule：向delayedTasks中添加定时任务
leastLoadNode：查找Kafka集群中负载最低的Node