kafka 生产者__sender线程源码剖析

kafka 生产者__sender线程源码剖析

前面我们已经讲解到kafka将消息放入消息累加器中，在本篇中将分析sender如何从RecordAccumulator中获取消息，并发送到broker去。

在实例化kafka对象的时候，我们会创建并启动sender线程：

           this.sender = newSender(logContext, kafkaClient, this.metadata);
            String ioThreadName = NETWORK_THREAD_PREFIX + " | " + clientId;
            this.ioThread = new KafkaThread(ioThreadName, this.sender, true);
            this.ioThread.start();

sender线程会在producer关闭前，一直循环调用 runOnce()方法：

 void runOnce() {
        if (transactionManager != null) {
			// ... 省略开启事务的一些操作
        }

        long currentTimeMs = time.milliseconds();
        long pollTimeout = sendProducerData(currentTimeMs);
        client.poll(pollTimeout, currentTimeMs);
    }

可以看出，该方法的核心代码是：sendProducerData(currentTimeMs);

以及 client.poll(pollTimeout, currentTimeMs);

• sendProducerData()方法解析：

  private long sendProducerData(long now) {
      	//1、通过生产者的相关源信息获取kafka集群
          Cluster cluster = metadata.fetch();
          //获取已经准备好的将要发送的数据
          RecordAccumulator.ReadyCheckResult result = this.accumulator.ready(cluster, now);
  
   		 // ...
  		
          // 2、create produce requests
        // 将要发送的消息转换成  <NodeId,List<ProducerBatch>>的形式，这里完成了从应用层到I/O层的转换 ，并从消息累加器中将这些ProducerBatch 移除
          Map<Integer, List<ProducerBatch>> batches = this.accumulator.drain(cluster, result.readyNodes, this.maxRequestSize, now);
      //3、将这些ProducerBatch加入到正在处理的  数据中  inFlightBatches
          addToInflightBatches(batches);
  		
      //  4、获取已经超时未发送的 消息  如果有就调用failBatch方法进行处理
          accumulator.resetNextBatchExpiryTime();
          List<ProducerBatch> expiredInflightBatches = getExpiredInflightBatches(now);
          List<ProducerBatch> expiredBatches = this.accumulator.expiredBatches(now);
          expiredBatches.addAll(expiredInflightBatches);
  
  		//5、执行请求发送
          sendProduceRequests(batches, now);
          return pollTimeout;
      }
  
  

  从上面的分析我们可以看出,sendProducerData()方法主要做了如下事情：

  • 1、调用消息累加器的ready（）方法获取到已经准备好的数据
  • 2、完成了消息由逻辑层的topicPartition到IO层的转换并将将要发送批次的消息从队列中移除。
  • 3、将准备要发送的数据加入到inflightBatches中
  • 4、调用sendProducerRequests()方法完成消息的发送。

  接下来我们来看sendProduceRequests方法完成的功能：

     private void sendProduceRequests(Map<Integer, List<ProducerBatch>> collated, long now) {
          for (Map.Entry<Integer, List<ProducerBatch>> entry : collated.entrySet())
              sendProduceRequest(now, entry.getKey(), acks, requestTimeoutMs, entry.getValue());
      }
  

  即循环调用sendProduceRequest()来进行处理。sendProduceRequest方法如下：

  主要完成的功能为：根据给定的消息批次创建一个ClientRequest请求对象，并调用NetworkClient的send方法来发送数据

    /**
       * Create a produce request from the given record batches
       */
      private void sendProduceRequest(long now, int destination, short acks, int timeout, List<ProducerBatch> batches) {
          	
               ProduceRequest.Builder requestBuilder = ProduceRequest.Builder.forMagic(minUsedMagic, acks, timeout,
                  produceRecordsByPartition, transactionalId);
          // 创建回调对象，当请求响应的时候，将会回调onComplete方法
             RequestCompletionHandler callback = new RequestCompletionHandler() {
              public void onComplete(ClientResponse response) {
                  handleProduceResponse(response, recordsByPartition, time.milliseconds());
              }
          };
   		
          //将batches 转换为requestBuilder  对象，并使用其创建clientRequest
          String nodeId = Integer.toString(destination);
          ClientRequest clientRequest = client.newClientRequest(nodeId, requestBuilder, now, acks != 0,
                  requestTimeoutMs, callback);
          //调用客户端的send方法发送数据
          client.send(clientRequest, now);
        
      }
  

  最后我们再次追踪下send方法的实现：

      @Override
      public void send(ClientRequest request, long now) {
          doSend(request, false, now);
      }
  

  那么 doSend(request, false, now);方法做了什么事情，我们不妨继续看看

   private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long now) {		//确认客户端时
          ensureActive();
          String nodeId = clientRequest.destination();
  		//获取到requestBuilder对象
          AbstractRequest.Builder<?> builder = clientRequest.requestBuilder();
          try {
              //
              NodeApiVersions versionInfo = apiVersions.get(nodeId);
     			//执行发送操作
              doSend(clientRequest, isInternalRequest, now, builder.build(version));
          } catch (UnsupportedVersionException unsupportedVersionException) {
              //进行异常处理
          }
  			
      }
  
  

  我们继续往下追doSend(clientRequest, isInternalRequest, now, builder.build(version)); 方法：

      private void doSend(ClientRequest clientRequest, boolean isInternalRequest, long now, AbstractRequest request) {
          String destination = clientRequest.destination();
          RequestHeader header = clientRequest.makeHeader(request.version());
  		//创建send实例
          Send send = request.toSend(destination, header);
          InFlightRequest inFlightRequest = new InFlightRequest(
                  clientRequest,
                  header,
                  isInternalRequest,
                  request,
                  send,
                  now);
          //这个是用来保存将要发送或等待响应集
          this.inFlightRequests.add(inFlightRequest);
          //将给定的请求入队，给后续的poll方法调用
          selector.send(send);
      }
  

• client.poll(pollTimeout, currentTimeMs);

  @Override
    public List<ClientResponse> poll(long timeout, long now) {
        ensureActive();

        if (!abortedSends.isEmpty()) {
            // If there are aborted sends because of unsupported version exceptions or disconnects,
            // handle them immediately without waiting for Selector#poll.
            List<ClientResponse> responses = new ArrayList<>();
            handleAbortedSends(responses);
            completeResponses(responses);
            return responses;
        }

        long metadataTimeout = metadataUpdater.maybeUpdate(now);
        try {
            //这个时使用selector来执行数据发送操作
            this.selector.poll(Utils.min(timeout, metadataTimeout, defaultRequestTimeoutMs));
        } catch (IOException e) {
            log.error("Unexpected error during I/O", e);
        }

        //  执行流程完成操作
        long updatedNow = this.time.milliseconds();
        List<ClientResponse> responses = new ArrayList<>();
        handleCompletedSends(responses, updatedNow);
        handleCompletedReceives(responses, updatedNow);
        handleDisconnections(responses, updatedNow);
        handleConnections();
        handleInitiateApiVersionRequests(updatedNow);
        handleTimedOutRequests(responses, updatedNow);
        completeResponses(responses);

        return responses;
    }

至此，我们已经完成整个kafka生产端的数据发送源码解析工作。