kafka副本机制（四）：ReplicaManager的定时任务

最新推荐文章于 2024-07-03 08:52:43 发布

其实系一个须刨

最新推荐文章于 2024-07-03 08:52:43 发布

阅读量436

点赞数

分类专栏： Kafka-0.10

本文链接：https://blog.csdn.net/lianggx3/article/details/99710916

版权

Kafka-0.10 专栏收录该内容

73 篇文章 14 订阅

订阅专栏

ReplicaManager总共有三个定时任务hw-checkpoint(周期性记录每个replica的hw，并保存到log目录中的replication-offset-checkpoint文件中)、isr-expiration(周期性调用maybeShrinkIsr()方法检测每个分区是否需要缩减ISR集合)和isr-change-propagation(周期性把ISR集合发生变化的分区记录到ZK中)三个任务

def startHighWaterMarksCheckPointThread() = {
    if(highWatermarkCheckPointThreadStarted.compareAndSet(false, true))
      scheduler.schedule("highwatermark-checkpoint", checkpointHighWatermarks, period = config.replicaHighWatermarkCheckpointIntervalMs, unit = TimeUnit.MILLISECONDS)
}
def checkpointHighWatermarks() {
    //获取全部的replica对象，按照副本所在的log目录进行分组
    val replicas = allPartitions.values.flatMap(_.getReplica(config.brokerId))
    val replicasByDir = replicas.filter(_.log.isDefined).groupBy(_.log.get.dir.getParentFile.getAbsolutePath)
    // 遍历所有的log目录
    for ((dir, reps) <- replicasByDir) {
        //收集当前log目录下的replication-offset-checkpoint文件
      val hwms = reps.map(r => new TopicAndPartition(r) -> r.highWatermark.messageOffset).toMap
      try {
        highWatermarkCheckpoints(dir).write(hwms)
      } catch {
        case e: IOException =>
          fatal("Error writing to highwatermark file: ", e)
          Runtime.getRuntime().halt(1)
      }
    }
}

isr-expiration周期性调用maybeShrinkIsr，前面已经介绍过，现在不做多余介绍
isr-change-propagation把isr变化记录到zk中，条件是isrChangeSet集合不为空，且最后一次又isr集合发生变化的时间距今已经超过5s，或者上次写入zk的时间距今已经超过了60s。

def maybePropagateIsrChanges() {
    val now = System.currentTimeMillis()
    isrChangeSet synchronized {
      if (isrChangeSet.nonEmpty &&
        (lastIsrChangeMs.get() + ReplicaManager.IsrChangePropagationBlackOut < now ||
          lastIsrPropagationMs.get() + ReplicaManager.IsrChangePropagationInterval < now)) {
        ReplicationUtils.propagateIsrChanges(zkUtils, isrChangeSet)
        isrChangeSet.clear()
        lastIsrPropagationMs.set(now)
      }
    }
}