hadoop 2.2.0 fsimage和edit logs的处理逻辑

最新推荐文章于 2022-09-25 16:29:50 发布

迷途小码

最新推荐文章于 2022-09-25 16:29:50 发布

阅读量1.7k

点赞数

CC 4.0 BY-SA版权

分类专栏： hadoop开发

本文链接：https://blog.csdn.net/knowledgeaaa/article/details/23864211

hadoop开发专栏收录该内容

56 篇文章

订阅专栏

本文详细介绍了Hadoop HDFS中Namenode如何处理fsimage和edit logs的逻辑，具体包括：确定可删除的最小txid，计算可删除的edit logs集合，并在集合超出限制时执行删除操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

在类org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager的purgeOldStorage()方法中描述了fsimage和edit logs的处理逻辑：

一、找到存在于fsimage中的最小txid，删除比最小txid小的fsimage

二、最小txid - dfs.namenode.num.extra.edits.retained = 可以删除txid集合

三、可删除txid集合 > dfs.namenode.max.extra.edits.segments.retained 时，删除集合中的最小值

完整代码：

  public void purgeOldStorage() throws IOException {
    FSImageTransactionalStorageInspector inspector =
      new FSImageTransactionalStorageInspector();
    storage.inspectStorageDirs(inspector);

    long minImageTxId = getImageTxIdToRetain(inspector);
    purgeCheckpointsOlderThan(inspector, minImageTxId);
    // If fsimage_N is the image we want to keep, then we need to keep
    // all txns > N. We can remove anything < N+1, since fsimage_N
    // reflects the state up to and including N. However, we also
    // provide a "cushion" of older txns that we keep, which is
    // handy for HA, where a remote node may not have as many
    // new images.
    //
    // First, determine the target number of extra transactions to retain based
    // on the configured amount.
    long minimumRequiredTxId = minImageTxId + 1;
    long purgeLogsFrom = Math.max(0, minimumRequiredTxId - numExtraEditsToRetain);
    
    ArrayList<EditLogInputStream> editLogs = new ArrayList<EditLogInputStream>();
    purgeableLogs.selectInputStreams(editLogs, purgeLogsFrom, false, false);
    Collections.sort(editLogs, new Comparator<EditLogInputStream>() {
      @Override
      public int compare(EditLogInputStream a, EditLogInputStream b) {
        return ComparisonChain.start()
            .compare(a.getFirstTxId(), b.getFirstTxId())
            .compare(a.getLastTxId(), b.getLastTxId())
            .result();
      }
    });

    // Remove from consideration any edit logs that are in fact required.
    while (editLogs.size() > 0 &&
        editLogs.get(editLogs.size() - 1).getFirstTxId() >= minimumRequiredTxId) {
      editLogs.remove(editLogs.size() - 1);
    }

    // Next, adjust the number of transactions to retain if doing so would mean
    // keeping too many segments around.
    while (editLogs.size() > maxExtraEditsSegmentsToRetain) {
      purgeLogsFrom = editLogs.get(0).getLastTxId() + 1;
      editLogs.remove(0);
    }
    
    // Finally, ensure that we're not trying to purge any transactions that we
    // actually need.
    if (purgeLogsFrom > minimumRequiredTxId) {
      throw new AssertionError("Should not purge more edits than required to "
          + "restore: " + purgeLogsFrom + " should be <= "
          + minimumRequiredTxId);
    }
    
    purgeableLogs.purgeLogsOlderThan(purgeLogsFrom);
  }