kafka源码解析之十一KafkaApis

最新推荐文章于 2024-03-17 15:22:47 发布

亮亮-AC米兰

最新推荐文章于 2024-03-17 15:22:47 发布

阅读量5.5k

点赞数

分类专栏： kafka 源码解析 kafka源码解析文章标签： kafka 源码

本文链接：https://blog.csdn.net/wl044090432/article/details/51068413

版权

本文详细探讨了KafkaApis中的11种请求处理，包括ProduceKey、FetchKey、OffsetsKey等，解析了请求来源和处理流程，重点关注了ProduceRequest的ack策略。

摘要由CSDN通过智能技术生成

KafkaApis真正负责内部request的业务逻辑，kafka_2.10-0.8.2.0目前支持11种request请求，具体的request类别在Broker处理的request的来源中已经提到，即下图：

接下来主要叙述下各个不同request的来源，以及request的处理流程：

class KafkaApis(val requestChannel: RequestChannel,
                val replicaManager: ReplicaManager,
                val offsetManager: OffsetManager,
                val zkClient: ZkClient,
                val brokerId: Int,
                val config: KafkaConfig,
                val controller: KafkaController) extends Logging {
def handle(request: RequestChannel.Request) {
  try{
    trace("Handling request: " + request.requestObj + " from client: " + request.remoteAddress)
    request.requestId match {
      case RequestKeys.ProduceKey => handleProducerOrOffsetCommitRequest(request)
      case RequestKeys.FetchKey => handleFetchRequest(request)
      case RequestKeys.OffsetsKey => handleOffsetRequest(request)//获取偏移量
      case RequestKeys.MetadataKey => handleTopicMetadataRequest(request)
      case RequestKeys.LeaderAndIsrKey => handleLeaderAndIsrRequest(request)
      case RequestKeys.StopReplicaKey => handleStopReplicaRequest(request)
      case RequestKeys.UpdateMetadataKey => handleUpdateMetadataRequest(request)
      case RequestKeys.ControlledShutdownKey => handleControlledShutdownRequest(request)
      case RequestKeys.OffsetCommitKey => handleOffsetCommitRequest(request)
      case RequestKeys.OffsetFetchKey => handleOffsetFetchRequest(request)
      case RequestKeys.ConsumerMetadataKey => handleConsumerMetadataRequest(request)
      case requestId => throw new KafkaException("Unknown api code " + requestId)
    }
  } catch {
    case e: Throwable =>
      request.requestObj.handleError(e, requestChannel, request)
      error("error when handling request %s".format(request.requestObj), e)
  } finally
    request.apiLocalCompleteTimeMs = SystemTime.milliseconds
}
}

11.1 RequestKeys.ProduceKey

来源：生成者发送消息至KAFKA集群/或者消费者提交偏移量至kafka集群，客户端根据分区函数把message发往属于该message的partition的leader的Broker。

def handleProducerOrOffsetCommitRequest(request: RequestChannel.Request) {
  val (produceRequest, offsetCommitRequestOpt) =
    if (request.requestId == RequestKeys.OffsetCommitKey) {//消费者提交偏移量至kafka集群，由参数offsets.storage决定，如果offsets.storage=kafka，则提交至kafka内部名为__consumer_offsets的topic的log
      val offsetCommitRequest = request.requestObj.asInstanceOf[OffsetCommitRequest]
      OffsetCommitRequest.changeInvalidTimeToCurrentTime(offsetCommitRequest)
      (producerRequestFromOffsetCommit(offsetCommitRequest), Some(offsetCommitRequest))
    } else {//正常的producer请求
      (request.requestObj.asInstanceOf[ProducerRequest], None)
    }
  if (produceRequest.requiredAcks > 1 || produceRequest.requiredAcks < -1) {
    warn(("Client %s from %s sent a produce request with request.required.acks of %d, which is now deprecated and will " +
      "be removed in next release. Valid values are -1, 0 or 1. Please consult Kafka documentation for supported " +
      "and recommended configuration.").format(produceRequest.clientId, request.remoteAddress, produceRequest.requiredAcks))
  }
  val sTime = SystemTime.milliseconds
//消息持久化
  val localProduceResults = appendToLocalLog(produceRequest, offsetCommitRequestOpt.nonEmpty)
  debug("Produce to local log in %d ms".format(SystemTime.milliseconds - sTime))
  val firstErrorCode = localProduceResults.find(_.errorCode != ErrorMapping.NoError).map(_.errorCode).getOrElse(ErrorMapping.NoError)
  val numPartitionsInError = localProduceResults.count(_.error.isDefined)
  if(produceRequest.requiredAcks == 0) {//acks == 0，即不需要ack，没啥需要特别做的
    // no operation needed if producer request.required.acks = 0; however, if there is any exception in handling the request, since
    // no response is expected by the producer the handler will send a close connection response to the socket server
    // to close the socket so that the producer client will know that some exception has happened and will refresh its metadata
    if (numPartitionsInError != 0) {//只有在错误的情况下才关闭conn使客户端感知到
      info(("Send the close connection response due to error handling produce request " +
        "[clientId = %s, correlationId = %s, topicAndPartition = %s] with Ack=0")
        .format(produceRequest.clientId, produceRequest.correlationId, produceRequest.topicPartitionMessageSizeMap.keySet.mkString(",")))
      requestChannel.closeConnection(request.processor, request)
    } else {
      if (firstErrorCode == ErrorMapping.NoError)//更新offsetsCache
        offsetCommitRequestOpt.foreach(ocr => offsetManager.putOffsets(ocr.groupId, ocr.requestInfo))
      if (offsetCommitRequestOpt.isDefined) {//但是如果是消费者提交偏移量至kafka集群的情况，则需要响应
        val response = offsetCommitRequestOpt.get.re