Kafka系列3----Log分析

Log是对segment的抽象，对多个segment的封装，外部只需要操作Log就可以了，不需要考虑往哪个segment读，往哪个segment写，内部会处理这些细节

看下Log的几个核心变量和函数：

    @volatile private var nextOffsetMetadata: LogOffsetMetadata = _

    private val segments: ConcurrentNavigableMap[java.lang.Long, LogSegment] = new ConcurrentSkipListMap[java.lang.Long, LogSegment]

    val topicPartition: TopicPartition = Log.parseTopicPartitionName(dir)

    private val tags = Map("topic" -> topicPartition.topic, "partition" -> topicPartition.partition.toString)
    @volatile var dir: File

    def logEndOffset: Long = nextOffsetMetadata.messageOffset

• segments：segment文件的集合，使用了ConcurrentSkipListMap类型(原理可以另行百度，这里不做详细分析)。key为baseOffset，即系列2里说的，例如000.log/index则baseOffset=0，238.log/index则baseOffset=238；value为Segment(后续分析)
• nextOffsetMetadata：主要记录message的offset，Segment的baseOffset，Segment的log的大小
• dir：Segment文件在磁盘上对应位置的File对象，即xxx/topic-artition/
• logEndOffset：nextOffsetMetadata的messageOffset的属性，即下一个消息的offset，也可以当做是当前最后的一个offset

segments的初始化

for (file <- dir.listFiles if file.isFile) {
    val filename = file.getName
    if (filename.endsWith(IndexFileSuffix) || filename.endsWith(TimeIndexFileSuffix)) {
        //segment index文件初始化....
    } else if (filename.endsWith(LogFileSuffix)) {
        //segment log文件初始化....
        segments.put(start, segment)
    }
}

启动的时候对partition下的segments进行遍历，填充segments

index初始化

    val logFile =
        if (filename.endsWith(TimeIndexFileSuffix))
            new File(file.getAbsolutePath.replace(TimeIndexFileSuffix, LogFileSuffix))
        else
            new File(file.getAbsolutePath.replace(IndexFileSuffix, LogFileSuffix))

    if (!logFile.exists) {
        warn("Found an orphaned index file, %s, with no corresponding log file.".format(file.getAbsolutePath))
        file.delete()
    }

没有做什么事情，主要是确保index对于的log文件存在

log初始化

//取得baseOffset，即238.log为238
val start = filename.substring(0, filename.length - LogFileSuffix.length).toLong
    val indexFile = Log.indexFilename(dir, start)
    val timeIndexFile = Log.timeIndexFilename(dir, start)

    val indexFileExists = indexFile.exists()
    val timeIndexFileExists = timeIndexFile.exists()
    //初始化Segment
    val segment = new LogSegment(dir = dir,startOffset = start,
        indexIntervalBytes = config.indexInterval,
        maxIndexSize = config.maxIndexSize,
        rollJitterMs = config.randomSegmentJitter,
        time = time, fileAlreadyExists = true)

    if (indexFileExists) {
        try {
  //索引文件检查
            segment.index.sanityCheck()
            if (!timeIndexFileExists)
                segment.timeIndex.resize(0)
            segment.timeIndex.sanityCheck()
        } catch {