Kafka 原理及代码分析

     // append an entry to the index (if needed)
      if (bytesSinceLastIndexEntry > indexIntervalBytes) {
        offsetIndex.append(largestOffset, physicalPosition)
        timeIndex.maybeAppend(maxTimestampSoFar, offsetOfMaxTimestampSoFar)
        bytesSinceLastIndexEntry = 0
      }
      bytesSinceLastIndexEntry += records.sizeInBytes

read():

入参：

startOffset: Long：

读取的第一个消息的位移

maxSize: Int：

能读取的最大字节数

maxPosition: Long = size：

能读取日志中能读取的最大位置

minOneMessage: Boolean = false：

是否返回至少一条消息，当读取的消息大于maxSize时

流程：

1.根据startOffset通过translateOffset，找到要读取的其实文件位置。

2.计算要读取的文件大小。

    val adjustedMaxSize =
      if (minOneMessage) math.max(maxSize, startOffsetAndSize.size)
      else maxSize
    if (adjustedMaxSize == 0)
      return FetchDataInfo(offsetMetadata, MemoryRecords.EMPTY)
    val fetchSize: Int = min((maxPosition - startPosition).toInt, adjustedMaxSize)

3.通过FileRecords.slice读取指定位置指定大小的数据

recover():

参数：

producerStateManager: ProducerStateManager：
leaderEpochCache: Option[LeaderEpochFileCache]：

流程：

1.清空所有索引

    offsetIndex.reset()
    timeIndex.reset()
    txnIndex.reset()

2.遍历日志段中所有消息。

1.检查消息的位移是否合法。

2.更新日志段的最大时间戳及其所属消息的位移。

3.更新索引资源。

4.更新消息总字节数

5.更新producerStateManager和leaderEpochCache状态。

3.进行截断操作，将大于遍历所有消息获取的总大小以外的数据截取。

2.Log

变量：

保存了所有的需要使用的文件类型

 /** a log file */
  val LogFileSuffix = ".log"
  /** an index file */
  val IndexFileSuffix = ".index"
  /** a time index file */
  val TimeIndexFileSuffix = ".timeindex"
  val ProducerSnapshotFileSuffix = ".snapshot"
  /** an (aborted) txn index */
  val TxnIndexFileSuffix = ".txnindex"
  /** a file that is scheduled to be deleted */
  val DeletedFileSuffix = ".deleted"
  /** A temporary file that is being used for log cleaning */
  val CleanedFileSuffix = ".cleaned"
  /** A temporary file used when swapping files into the log */
  val SwapFileSuffix = ".swap"
  /** Clean shutdown file that indicates the broker was cleanly shutdown in 0.8 and higher.
   * This is used to avoid unnecessary recovery after a clean shutdown. In theory this could be
   * avoided by passing in the recovery point, however finding the correct position to do this
   * requires accessing the offset index which may not be safe in an unclean shutdown.
   * For more information see the discussion in PR#2104
   */
  val CleanShutdownFile = ".kafka_cleanshutdown"
  /** a directory that is scheduled to be deleted */
  val DeleteDirSuffix = "-delete"
  /** a directory that is used for future partition */
  val FutureDirSuffix = "-future"

初始化：

  locally {
    initializePartitionMetadata()
    updateLogStartOffset(logStartOffset)
    maybeIncrementFirstUnstableOffset()
    initializeTopicId()
  }

initializePartitionMetadata()：
        创建分区日志路径。
updateLogStartOffset(logStartOffset)：
        设置高水位和回复点。
maybeIncrementFirstUnstableOffset()：
        First Unstable Offset事务机制的一部分
initializeTopicId()：
        初始topic id

方法：

appendAsFollower():

Followert添加日志

appendAsLeader():

Leader添加日志

2.请求处理

KafkaServer::startup():

初始化SocketServer
socketServer = new SocketServer(config, metrics, time, credentialProvider, apiVersionManager)
socketServer.startup(startProcessingRequests = false)

初始化数据面KafkaRequestHandlerPool
 dataPlaneRequestHandlerPool = new KafkaRequestHandlerPool(config.brokerId, socketServer.dataPlaneRequestChannel, dataPlaneRequestProcessor, time,
          config.numIoThreads, s"${SocketServer.DataPlaneMetricPrefix}RequestHandlerAvgIdlePercent", SocketServer.DataPlaneThreadPrefix)

初始化控制面KafkaRequestHandlerPool
 controlPlaneRequestHandlerPool = new KafkaRequestHandlerPool(config.brokerId, socketServer.controlPlaneRequestChannelOpt.get, controlPlaneRequestProcessor, time,
            1, s"${SocketServer.ControlPlaneMetricPrefix}RequestHandlerAvgIdlePercent", SocketServer.ControlPlaneThreadPrefix)

1.SocketServer

方法：

startup():
启动服务。
1.创建控制面的acceptor和processor线程。
2.创建数据面的acceptor和processor线程。
3.启动数据面和控制面的线程。

1.Acceptor

作用：

负责监听新接入的连接，确认有新连接接入后，将连接传递给某个processor处理线程(currentProcessorIndex = currentProcessorIndex % processors.length)

2.Processor

作用：

负责接收acceptor监听到的连接，并进行数据接收，接收数据之后通过

RequestChannel发送给KafkaRequestHandler线程进行处理。

成员：

newConnections:ArrayBlockingQueue：

将acceptor分配的新连接保存起来，等待处理。防止processor处理过慢，影响acceptor分配新连接。

2.KafkaRequestHandlerPool

作用:

管理线程池，每个线程执行KafkaRequestHandler方法，处理实际请求信息。具体处理有KafkaApis.handle()进行处理。

方法：

KafkaRequestHandlerPool():根据传入的numThreads创建对应数量的KafkaRequestHandler线程。
数据面有numThreads个处理消息的线程，控制面有一个处理消息的线程。

3.KafkaApis

作用:

处理收到的实际请求内容。

处理的所有消息：

 case ApiKeys.PRODUCE => handleProduceRequest(request, requestLocal)
        case ApiKeys.FETCH => handleFetchRequest(request)
        case ApiKeys.LIST_OFFSETS => handleListOffsetRequest(request)
        case ApiKeys.METADATA => handleTopicMetadataRequest(request)
        case ApiKeys.LEADER_AND_ISR => handleLeaderAndIsrRequest(request)
        case ApiKeys.STOP_REPLICA => handleStopReplicaRequest(request)
        case ApiKeys.UPDATE_METADATA => handleUpdateMetadataRequest(request, requestLocal)
        case ApiKeys.CONTROLLED_SHUTDOWN => handleControlledShutdownRequest(request)
        case ApiKeys.OFFSET_COMMIT => handleOffsetCommitRequest(request, requestLocal)
        case ApiKeys.OFFSET_FETCH => handleOffsetFetchRequest(request)
        case ApiKeys.FIND_COORDINATOR => handleFindCoordinatorRequest(request)
        case ApiKeys.JOIN_GROUP => handleJoinGroupRequest(request, requestLocal)
        case ApiKeys.HEARTBEAT => handleHeartbeatRequest(request)
        case ApiKeys.LEAVE_GROUP => handleLeaveGroupRequest(request)
        case ApiKeys.SYNC_GROUP => handleSyncGroupRequest(request, requestLocal)
        case ApiKeys.DESCRIBE_GROUPS => handleDescribeGroupRequest(request)
        case ApiKeys.LIST_GROUPS => handleListGroupsRequest(request)
        case ApiKeys.SASL_HANDSHAKE => handleSaslHandshakeRequest(request)
        case ApiKeys.API_VERSIONS => handleApiVersionsRequest(request)
        case ApiKeys.CREATE_TOPICS => maybeForwardToController(request, handleCreateTopicsRequest)
        case ApiKeys.DELETE_TOPICS => maybeForwardToController(request, handleDeleteTopicsRequest)
        case ApiKeys.DELETE_RECORDS => handleDeleteRecordsRequest(request)
        case ApiKeys.INIT_PRODUCER_ID => handleInitProducerIdRequest(request, requestLocal)
        case ApiKeys.OFFSET_FOR_LEADER_EPOCH => handleOffsetForLeaderEpochRequest(request)
        case ApiKeys.ADD_PARTITIONS_TO_TXN => handleAddPartitionToTxnRequest(request, requestLocal)
        case ApiKeys.ADD_OFFSETS_TO_TXN => handleAddOffsetsToTxnRequest(request, requestLocal)
        case ApiKeys.END_TXN => handleEndTxnRequest(request, requestLocal)
        case ApiKeys.WRITE_TXN_MARKERS => handleWriteTxnMarkersRequest(request, requestLocal)
        case ApiKeys.TXN_OFFSET_COMMIT => handleTxnOffsetCommitRequest(request, requestLocal)
        case ApiKeys.DESCRIBE_ACLS => handleDescribeAcls(request)
        case ApiKeys.CREATE_ACLS => maybeForwardToController(request, handleCreateAcls)
        case ApiKeys.DELETE_ACLS => maybeForwardToController(request, handleDeleteAcls)
        case ApiKeys.ALTER_CONFIGS => maybeForwardToController(request, handleAlterConfigsRequest)
        case ApiKeys.DESCRIBE_CONFIGS => handleDescribeConfigsRequest(request)
        case ApiKeys.ALTER_REPLICA_LOG_DIRS => handleAlterReplicaLogDirsRequest(request)
        case ApiKeys.DESCRIBE_LOG_DIRS => handleDescribeLogDirsRequest(request)
        case ApiKeys.SASL_AUTHENTICATE => handleSaslAuthenticateRequest(request)
        case ApiKeys.CREATE_PARTITIONS => maybeForwardToController(request, handleCreatePartitionsRequest)
        case ApiKeys.CREATE_DELEGATION_TOKEN => maybeForwardToController(request, handleCreateTokenRequest)
        case ApiKeys.RENEW_DELEGATION_TOKEN => maybeForwardToController(request, handleRenewTokenRequest)
        case ApiKeys.EXPIRE_DELEGATION_TOKEN => maybeForwardToController(request, handleExpireTokenRequest)
        case ApiKeys.DESCRIBE_DELEGATION_TOKEN => handleDescribeTokensRequest(request)
        case ApiKeys.DELETE_GROUPS => handleDeleteGroupsRequest(request, requestLocal)
        case ApiKeys.ELECT_LEADERS => handleElectReplicaLeader(request)
        case ApiKeys.INCREMENTAL_ALTER_CONFIGS => maybeForwardToController(request, handleIncrementalAlterConfigsRequest)
        case ApiKeys.ALTER_PARTITION_REASSIGNMENTS => maybeForwardToController(request, handleAlterPartitionReassignmentsRequest)
        case ApiKeys.LIST_PARTITION_REASSIGNMENTS => maybeForwardToController(request, handleListPartitionReassignmentsRequest)
        case ApiKeys.OFFSET_DELETE => handleOffsetDeleteRequest(request, requestLocal)
        case ApiKeys.DESCRIBE_CLIENT_QUOTAS => handleDescribeClientQuotasRequest(request)
        case ApiKeys.ALTER_CLIENT_QUOTAS => maybeForwardToController(request, handleAlterClientQuotasRequest)
        case ApiKeys.DESCRIBE_USER_SCRAM_CREDENTIALS => handleDescribeUserScramCredentialsRequest(request)
        case ApiKeys.ALTER_USER_SCRAM_CREDENTIALS => maybeForwardToController(request, handleAlterUserScramCredentialsRequest)
        case ApiKeys.ALTER_ISR => handleAlterIsrRequest(request)
        case ApiKeys.UPDATE_FEATURES => maybeForwardToController(request, handleUpdateFeatures)
        case ApiKeys.ENVELOPE => handleEnvelope(request, requestLocal)
        case ApiKeys.DESCRIBE_CLUSTER => handleDescribeCluster(request)
        case ApiKeys.DESCRIBE_PRODUCERS => handleDescribeProducersRequest(request)
        case ApiKeys.DESCRIBE_TRANSACTIONS => handleDescribeTransactionsRequest(request)
        case ApiKeys.LIST_TRANSACTIONS => handleListTransactionsRequest(request)
        case ApiKeys.ALLOCATE_PRODUCER_IDS => handleAllocateProducerIdsRequest(request)
        case ApiKeys.DESCRIBE_QUORUM => forwardToControllerOrFail(request)

3.Controller

1.ControllerContext

作用：

元数据信息。

成员：

最低0.47元/天解锁文章

8ZERO8

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
Kafka 原理及代码分析

Kafka版本3.0.01.LogSegmentappend():如参：startOffset: Long：maxSize: Int：maxPosition: Long = size：minOneMessage: Boolean = false：read():recover():
复制链接

扫一扫