hadoop 2.6.0 hadoop 对Namenode image文件和Edit文件进行清理的NNStorageRetentionManager源代码分析

最新推荐文章于 2023-11-29 22:30:10 发布

houzhizhen

最新推荐文章于 2023-11-29 22:30:10 发布

阅读量1.5k

点赞数

分类专栏： hadoop-hdfs

本文链接：https://blog.csdn.net/houzhizhen/article/details/50595076

版权

hadoop-hdfs 专栏收录该内容

40 篇文章 0 订阅

订阅专栏

NNStorageRetentionManager对文件进行定期旋转删除。

NNStorageRetentionManager的构造方法如下:

  public NNStorageRetentionManager(Configuration conf, NNStorage storage,
      LogsPurgeable purgeableLogs) {
    this(conf, storage, purgeableLogs, new DeletionStoragePurger());
  }

构造NNStorageRetentionManager对象的时候，生成一个DeletionStoragePurger对象，此对象有purgeImage(FSImageFile)来清理Image文件，purgeLog(EditLogFile)来清理EditLog文件，在deleteOrWarn方法里，如果删除不成功，会打印一个警告日志，代码如下：

static class DeletionStoragePurger implements StoragePurger {
    @Override
    public void purgeLog(EditLogFile log) {
      LOG.info("Purging old edit log " + log);
      deleteOrWarn(log.getFile());
    }

    @Override
    public void purgeImage(FSImageFile image) {
      LOG.info("Purging old image " + image);
      deleteOrWarn(image.getFile());
      deleteOrWarn(MD5FileUtils.getDigestFileForFile(image.getFile()));
    }

    private static void deleteOrWarn(File file) {
      if (!file.delete()) {
        // It's OK if we fail to delete something -- we'll catch it
        // next time we swing through this directory.
        LOG.warn("Could not delete " + file);
      }      
    }
  }

DeletionStoragePurger实现了StoragePurger方法，

  static interface StoragePurger {
    void purgeLog(EditLogFile log);
    void purgeImage(FSImageFile image);
  }

NNStorageRetentionManager的构造方法又调用了以下构造方法：

public NNStorageRetentionManager(
      Configuration conf,
      NNStorage storage,
      LogsPurgeable purgeableLogs,
      StoragePurger purger) {
    this.numCheckpointsToRetain = conf.getInt(
        DFSConfigKeys.DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_KEY,
        DFSConfigKeys.DFS_NAMENODE_NUM_CHECKPOINTS_RETAINED_DEFAULT);
    this.numExtraEditsToRetain = conf.getLong(
        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY,
        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_DEFAULT);
    this.maxExtraEditsSegmentsToRetain = conf.getInt(
        DFSConfigKeys.DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_KEY,
        DFSConfigKeys.DFS_NAMENODE_MAX_EXTRA_EDITS_SEGMENTS_RETAINED_DEFAULT);
    Preconditions.checkArgument(numCheckpointsToRetain > 0,
        "Must retain at least one checkpoint");
    Preconditions.checkArgument(numExtraEditsToRetain >= 0,
        DFSConfigKeys.DFS_NAMENODE_NUM_EXTRA_EDITS_RETAINED_KEY +
        " must not be negative");
    
    this.storage = storage;
    this.purgeableLogs = purgeableLogs;
    this.purger = purger;
  }

在每次调用saveFSImageInAllDirs的时候，就会调用一次purgeOldStorage()方法。