HDFS源代码分析之DataNode DirectoryScanner实现

DirectoryScanner

DirectoryScanner的主要任务是定期扫描磁盘上的数据块，检查磁盘上的数据块信息是否与FsDatasetImpl中保存的数据块信息一致，如果不一致则对FsDatasetImpl中的信息进行更新。

注：DirectoryScanner只检查内存和磁盘上FINALIZED状态的数据块是否一致。

public class DirectoryScanner implements Runnable {
  ...
  // 异步收集磁盘数据块信息的线程池
  private final ExecutorService reportCompileThreadPool;
  // master线程,定期调用DirectoryScanner.run方法
  private final ScheduledExecutorService masterThread;
  // 数据块信息与内存元数据的差异,扫描过程中更新,扫描结束后把diffs更新到FsDatasetImpl
  final ScanInfoPerBlockPool diffs = new ScanInfoPerBlockPool();
  ...

DirectoryScanner(DataNode datanode, FsDatasetSpi<?> dataset, Configuration conf) {
  ...
    reportCompileThreadPool = Executors.newFixedThreadPool(threads, 
        new Daemon.DaemonFactory());
    //初始化master线程
    masterThread = new ScheduledThreadPoolExecutor(1,
        new Daemon.DaemonFactory());
  }

void start() {
    ...
    // 定期执行DirectoryScanner的run方法
    masterThread.scheduleAtFixedRate(this, offset, scanPeriodMsecs, 
                                     TimeUnit.MILLISECONDS);
  }

  @Override
  public void run() {
    try {
      if (!shouldRun) {
        //shutdown has been activated
        LOG.warn("this cycle terminating immediately because 'shouldRun' has been deactivated");
        return;
      }

      //We're are okay to run - do it
      reconcile();      
      
    } catch (Exception e) {
      //Log and continue - allows Executor to run again next cycle
      LOG.error("Exception during DirectoryScanner execution - will continue next cycle", e);
    } catch (Error er) {
      //Non-recoverable error - re-throw after logging the problem
      LOG.error("System Error during DirectoryScanner execution - permanently terminating periodic scanner", er);
      throw er;
    }
  }

  void reconcile() throws IOException {
    // 调用scan搜集磁盘数据块与内存数据块的差异信息
    scan();
    for (Entry<String, LinkedList<ScanInfo>> entry : diffs.entrySet()) {
      String bpid = entry.getKey();
      LinkedList<ScanInfo> diff = entry.getValue();
      
      for (ScanInfo info : diff) {
        // 与FsDatasetImpl完成同步
        dataset.checkAndUpdate(bpid, info.getBlockId(), info.getBlockFile(),
            info.getMetaFile(), info.getVolume());
      }
    }
    if (!retainDiffs) clear();
  }
}

DirectoryScanner对象会定期在master线程上出发扫描任务，这个任务由reconcile()方法完成。