Kafka服务端源码剖析四 -- Processor线程将接受到的请求加入到stagedReceives队列、completedReceives队列

上一节我们分析到Acceptor把接收到的请求给了不同的Processor，然而Processor只是把这些请求给缓存起来了。这些请求终归是需要处理的，接下来我们就看Processor是如何处理这些请求的？，首先从run（）方法开始：

  override def run() {
    startupComplete()
    while (isRunning) {
      try {
        // setup any new connections that have been queued up
        //TODO 注册 OP_READ事件
        configureNewConnections()
        // register any new responses for writing
        processNewResponses()
        //TODO poll 处理读请求，把请求存入了stagedReceives,completedReceives
        poll()
        //TODO 处理已经接收到的请求
        processCompletedReceives()
        processCompletedSends()
        processDisconnected()
      } catch {
        // We catch all the throwables here to prevent the processor thread from exiting. We do this because
        // letting a processor exit might cause a bigger impact on the broker. Usually the exceptions thrown would
        // be either associated with a specific socket channel or a bad request. We just ignore the bad socket channel
        // or request. This behavior might need to be reviewed if we see an exception that need the entire broker to stop.
        case e: ControlThrowable => throw e
        case e: Throwable =>
          error("Processor got uncaught exception.", e)
      }
    }

    debug("Closing selector - processor " + id)
    swallowError(closeAll())
    shutdownComplete()
  }

我们接着看 configureNewConnections() 方法是怎么注册OP_READ事件的：

/**
   * Register any new connections that have been queued up
   */
  private def configureNewConnections() {
    //上一节我们将客户端过来的连接加入到队列newConnections中
    //队列newConnections不为空的时候，不断迭代队列里面的连接
    while (!newConnections.isEmpty) {
      //TODO 从队列里面获取channel
      val channel = newConnections.poll()
      try {
        debug(s"Processor $id listening to new connection from ${channel.socket.getRemoteSocketAddress}")
        val localHost = channel.socket().getLocalAddress.getHostAddress
        val localPort = channel.socket().getLocalPort
        val remoteHost = channel.socket().getInetAddress.getHostAddress
        val remotePort = channel.socket().getPort
        val connectionId = ConnectionId(localHost, localPort, remoteHost, remotePort).toString
        //TODO 把channel注册到Selector上，注册监听OP_READ事件
        selector.register(connectionId, channel)
      } catch {
        // We explicitly catch all non fatal exceptions and close the socket to avoid a socket leak. The other
        // throwables will be caught in processor and logged as uncaught exceptions.
        case NonFatal(e) =>
          val remoteAddress = channel.getRemoteAddress
          // need to close the channel here to avoid a socket leak.
          close(channel)
          error(s"Processor $id closed connection from $remoteAddress", e)
      }
    }
  }

接着看 selector.register(connectionId, channel)，这里才是真正实现读取事件的注册，如下：

/**
     * Register the nioSelector with an existing channel
     * Use this on server-side, when a connection is accepted by a different thread but processed by the Selector
     * Note that we are not checking if the connection id is valid - since the connection already exists
     */
    public void register(String id, SocketChannel socketChannel) throws ClosedChannelException {
        //TODO 真正的注册操作,注册了OP_READ事件，这样就可以读取发送过来的信息了。
        SelectionKey key = socketChannel.register(nioSelector, SelectionKey.OP_READ);
        KafkaChannel channel = channelBuilder.buildChannel(id, key, maxReceiveSize);
        key.attach(channel);
        //把channel放到channels
        this.channels.put(id, channel);
    }

我再看poll方法是怎么将连接请求存入了stagedReceives的？

private def poll() {
    //TODO 调用Selector的poll方法
    try selector.poll(300)
    catch {
      case e @ (_: IllegalStateException | _: IOException) =>
        error(s"Closing processor $id due to illegal state or IO exception")
        swallow(closeAll())
        shutdownComplete()
        throw e
    }
  }

点击selector.poll(300):

@Override
    public void poll(long timeout) throws IOException {
        if (timeout < 0)
            throw new IllegalArgumentException("timeout should be >= 0");
        clear();

        if (hasStagedReceives() || !immediatelyConnectedKeys.isEmpty())
            timeout = 0;

        /* check ready keys */
        long startSelect = time.nanoseconds();
        //TODO 执行select 操作，返回key
        int readyKeys = select(timeout);
        long endSelect = time.nanoseconds();
        this.sensors.selectTime.record(endSelect - startSelect, time.milliseconds());

        if (readyKeys > 0 || !immediatelyConnectedKeys.isEmpty()) {
            //TODO 对所有绑定在Selector上的key进行处理 （一个broker连接就对应一个key）
            pollSelectionKeys(this.nioSelector.selectedKeys(), false, endSelect);
            pollSelectionKeys(immediatelyConnectedKeys, true, endSelect);
        }

        addToCompletedReceives();

        long endIo = time.nanoseconds();
        this.sensors.ioTime.record(endIo - endSelect, time.milliseconds());

        // we use the time at the end of select to ensure that we don't close any connections that
        // have just been processed in pollSelectionKeys
        maybeCloseOldestConnection(endSelect);
    }

我们主要关注这个方法 pollSelectionKeys()，该方法在生产者源码中已经有详细的说明，我这里只看部分关键代码，如下：

    private void pollSelectionKeys(Iterable<SelectionKey> selectionKeys,
                                   boolean isImmediatelyConnected,
                                   long currentTimeNanos) {
        //TODO 遍历每一个SelectionKey
        Iterator<SelectionKey> iterator = selectionKeys.iterator();
        while (iterator.hasNext()) {
            SelectionKey key = iterator.next();
            iterator.remove();
            //TODO 根据key获取到对应的KafkaChannel
            KafkaChannel channel = channel(key);

            // register all per-connection metrics at once
            sensors.maybeRegisterConnectionMetrics(channel.id());
            if (idleExpiryManager != null)
                idleExpiryManager.update(channel.id(), currentTimeNanos);

            try {
            
                略  . . . . . . 
               
                /* if channel is ready read from any connections that have readable data */
                //TODO 重要代码
                //这儿处理的是NIO里读请求的事件，那很明显
                //这个方法会在broker返回了一些服务端的响应的时候执行的。
                //只有key绑定了READ事件才能执行这个方法
                //但是目前key没有绑定READ事件
                if (channel.ready() && key.isReadable() && !hasStagedReceive(channel)) {
                    NetworkReceive networkReceive;
                    //TODO 读取数据
                    // 这个地方读取响应的时候可以会读取到多个响应。所以里面的代码需要处理一个粘包的问题。
                    while ((networkReceive = channel.read()) != null)
                        addToStagedReceives(channel, networkReceive);
                }
                
             略  . . . . . . 

            } catch (Exception e) {
                String desc = channel.socketDescription();
                if (e instanceof IOException)
                    log.debug("Connection with {} disconnected", desc, e);
                else
                    log.warn("Unexpected error from {}; closing connection", desc, e);
                close(channel);
                this.disconnected.add(channel.id());
            }
        }
    }

继续点击 addToStagedReceives,发现方法将连接请求加入到 stagedReceives里面：

/**
     * adds a receive to staged receives
     */
    private void addToStagedReceives(KafkaChannel channel, NetworkReceive receive) {
        if (!stagedReceives.containsKey(channel))
            stagedReceives.put(channel, new ArrayDeque<NetworkReceive>());

        Deque<NetworkReceive> deque = stagedReceives.get(channel);
        deque.add(receive);
    }

我们再回到Selector的poll方法，代码继续往下执行到 addToCompletedReceives，点击 addToCompletedReceives，该方法就是将请求加入到已完成队列，如下：

/**
     * checks if there are any staged receives and adds to completedReceives
     */
    private void addToCompletedReceives() {
        //如果里面有数据
        if (!this.stagedReceives.isEmpty()) {
            //获取所有数据
            Iterator<Map.Entry<KafkaChannel, Deque<NetworkReceive>>> iter = this.stagedReceives.entrySet().iterator();
            //对数据进行遍历
            while (iter.hasNext()) {
                //每次遍历的是对应一个连接的数据。一个连接对应一个Deque
                Map.Entry<KafkaChannel, Deque<NetworkReceive>> entry = iter.next();
                KafkaChannel channel = entry.getKey();
                if (!channel.isMute()) {
                    Deque<NetworkReceive> deque = entry.getValue();
                    //获取一个连接的一个Deque里面的第一条消息
                    NetworkReceive networkReceive = deque.poll();
                    //把消息加入到completedReceives集合里面
                    this.completedReceives.add(networkReceive);
                    this.sensors.recordBytesReceived(channel.id(), networkReceive.payload().limit());
                    //队列读完了以后就移除
                    if (deque.isEmpty())
                        iter.remove();
                }
            }
        }
    }

总结：我们发现Processor线程并没有直接处理请求，而是直接将newConnections队列的请求加入到stagedReceives队列再加入到completedReceives队列。