YARN源码剖析：NM子服务初始化

最新推荐文章于 2024-07-11 22:23:39 发布
最新推荐文章于 2024-07-11 22:23:39 发布
阅读量862 收藏 1
点赞数
分类专栏： YARN 分布式计算
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_44630798/article/details/88769802
版权

NM初始化过程中涉及到的子service列表

  • DeletionService 文件清理服务
  • NodeHealthCheckerService 节点健康检查服务
  • NodeResourceMonitor 节点资源监控服务
  • ContainerManager 容器管理服务
    –ResourceLocalizationService 资源本地服务
    –ContainersLauncher container启动服务
    –ContianerMonitor container监控服务
    –AuxServices NM附属服务（如shuffle服务）
    –LogHandler（NonAggregatingLogHandler | LogAggregationService） NM本地log处理服务
    –SharedCacheUploadService
  • WebServer web服务
  • AsyncDispatcher 事件分发器
  • NodeStatusUpdater 节点状态更新服务
  • NMStateStoreService NM状态持久化服务

子service初始化分析

DeletionService

作用：删除本地文件

@Override
protected void serviceInit(Configuration conf) throws Exception {
//构建线程工厂
  ThreadFactory tf = new ThreadFactoryBuilder()
    .setNameFormat("DeletionService #%d")
    .build();
  if (conf != null) {
    //创建线程池，默认4个线程
    sched = new DelServiceSchedThreadPoolExecutor(
        conf.getInt(YarnConfiguration.NM_DELETE_THREAD_COUNT,
        YarnConfiguration.DEFAULT_NM_DELETE_THREAD_COUNT), tf);
    debugDelay = conf.getInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 0);
  } else {
    sched = new DelServiceSchedThreadPoolExecutor(
        YarnConfiguration.DEFAULT_NM_DELETE_THREAD_COUNT, tf);
  }
  sched.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
  sched.setKeepAliveTime(60L, SECONDS);
  if (stateStore.canRecover()) {
    recover(stateStore.loadDeletionServiceState());
  }
  super.serviceInit(conf);
}

相关参数
yarn.nodemanager.delete.thread-count 默认4
yarn.nodemanager.delete.debug-delay-sec 默认0

NodeHealthCheckerService

作用：检查节点健康状况并汇报
在这里插入图片描述
包括两个子service：
（1）NodeHealthScriptRunner–使用配置的脚本进行健康检查
（2）LocalDirsHandlerService–检查磁盘健康状况
包括一个方法getHealthReport()：用来获取（1）（2）两service的健康检查结果

@Override
protected void serviceInit(Configuration conf) throws Exception {
  //判读是否有配置健康检查脚本，有的话将脚本检查服务添加到NM的服务列表中
  if (NodeHealthScriptRunner.shouldRun(conf)) { 
    nodeHealthScriptRunner = new NodeHealthScriptRunner();
    addService(nodeHealthScriptRunner);
  }
  addService(dirsHandler);
  super.serviceInit(conf);
}
NodeHealthScriptRunner

作用：使用配置的脚本来检查节点的健康状况并汇报结果给汇报者。
在这里插入图片描述
实现：使用Timer定时的去执行健康检查脚本，健康检查脚本封装为一个TimerTask，timer支持超时检查功能，如果超过设定的时间脚本没有执行完，则timer会kill脚本并返回结果为timeout。
serviceInit分析

/*
 * Method which initializes the values for the script path and interval time.
 */
@Override
protected void serviceInit(Configuration conf) throws Exception {
  this.conf = conf;
  this.nodeHealthScript = 
      conf.get(YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_PATH);
  this.intervalTime = conf.getLong(YarnConfiguration.NM_HEALTH_CHECK_INTERVAL_MS,
      YarnConfiguration.DEFAULT_NM_HEALTH_CHECK_INTERVAL_MS);
  this.scriptTimeout = conf.getLong(
      YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_TIMEOUT_MS,
      YarnConfiguration.DEFAULT_NM_HEALTH_CHECK_SCRIPT_TIMEOUT_MS);
  String[] args = conf.getStrings(YarnConfiguration.NM_HEALTH_CHECK_SCRIPT_OPTS,
      new String[] {});
  //NodeHealthMonitorExecutor是TimerTask的继承类，即创建了一个TimerTask
  timer = new NodeHealthMonitorExecutor(args);
  super.serviceInit(conf);
}

相关参数
yarn.nodemanager.health-checker.script.path 脚本路径
yarn.nodemanager.health-checker.interval-ms 健康检查脚本执行间隔时间，默认10分钟
yarn.nodemanager.health-checker.script.timeout-ms 监控检查脚本执行超时时间，默认2*yarn.nodemanager.health-checker.interval-ms
yarn.nodemanager.health-checker.script.opts 给脚本传递的参数

LocalDirsHandlerService

作用：检查节点的本地目录健康状况，周期性的检查nodemanager-local-dirs和nodemanager-log-dirs两种配置目录。
实现：使用Timer定期去检查两个目录的使用率和是否为只读模式了

/**
   * Method which initializes the timertask and its interval time.
   * 
   */
  @Override
  protected void serviceInit(Configuration config) throws Exception {
    // Clone the configuration as we may do modifications to dirs-list
    Configuration conf = new Configuration(config);
    diskHealthCheckInterval = conf.getLong(
        YarnConfiguration.NM_DISK_HEALTH_CHECK_INTERVAL_MS,
        YarnConfiguration.DEFAULT_NM_DISK_HEALTH_CHECK_INTERVAL_MS);
    //真正的健康检查TimerTask实例
    monitoringTimerTask = new MonitoringTimerTask(conf);
    isDiskHealthCheckerEnabled = conf.getBoolean(
        YarnConfiguration.NM_DISK_HEALTH_CHECK_ENABLE, true);
    minNeededHealthyDisksFactor = conf.getFloat(
        YarnConfiguration.NM_MIN_HEALTHY_DISKS_FRACTION,
        YarnConfiguration.DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION);
    lastDisksCheckTime = System.currentTimeMillis();
    super.serviceInit(conf);

    //创建本地文件上下文实例
    FileContext localFs;
    try {
      localFs = FileContext.getLocalFSFileContext(config);
    } catch (IOException e) {
      throw new YarnRuntimeException("Unable to get the local filesystem", e);
    }
    //创建出数据目录和log目录
    FsPermission perm = new FsPermission((short)0755);
    boolean createSucceeded = localDirs.createNonExistentDirs(localFs, perm);
    createSucceeded &= logDirs.createNonExistentDirs(localFs, perm);
    if (!createSucceeded) {
      updateDirsAfterTest();
    }

    // Check the disk health immediately to weed out bad directories
    // before other init code attempts to use them.
    //执行一次目录健康检查
    checkDirs();
  }

MonitoringTimerTask

public MonitoringTimerTask(Configuration conf) throws YarnRuntimeException {
      float maxUsableSpacePercentagePerDisk =
          conf.getFloat(
            YarnConfiguration.NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE,
            YarnConfiguration.DEFAULT_NM_MAX_PER_DISK_UTILIZATION_PERCENTAGE);
      long minFreeSpacePerDiskMB =
          conf.getLong(YarnConfiguration.NM_MIN_PER_DISK_FREE_SPACE_MB,
            YarnConfiguration.DEFAULT_NM_MIN_PER_DISK_FREE_SPACE_MB);
      localDirs =
          new DirectoryCollection(
            validatePaths(conf
              .getTrimmedStrings(YarnConfiguration.NM_LOCAL_DIRS)),
            maxUsableSpacePercentagePerDisk, minFreeSpacePerDiskMB);
      logDirs =
          new DirectoryCollection(
            validatePaths(conf.getTrimmedStrings(YarnConfiguration.NM_LOG_DIRS)),
            maxUsableSpacePercentagePerDisk, minFreeSpacePerDiskMB);
      localDirsAllocator = new LocalDirAllocator(
          YarnConfiguration.NM_LOCAL_DIRS);
      logDirsAllocator = new LocalDirAllocator(YarnConfiguration.NM_LOG_DIRS);
    }

相关参数：
yarn.nodemanager.disk-health-checker.interval-ms 磁盘检查间隔时间，默认：2 * 60 * 1000
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage 最大磁盘使用率，默认90%
yarn.nodemanager.disk-health-checker.min-free-space-per-disk-mb 最小可用空间，默认0
yarn.nodemanager.local-dirs 本地数据目录，默认/tmp/nm-local-dir
yarn.nodemanager.log-dirs 本地log目录，默认/tmp/logs
yarn.nodemanager.disk-health-checker.enable 是否开始磁盘检查，默认true
min-healthy-disks 最小健康磁盘，最少的健康磁盘个数占比，小于这个比例的化NM就不能启动container了，默认25%

NodeResourceMonitor

NodeResourceMonitorImp是个空服务，没有任何实现

ContainerManager

作用：container管理器
子服务：
（1）资源本地服务ResourceLocalizationService
（2）container启动服务ContainersLauncher
（3）container监控服务ContianerMonitor
（4）配置服务AuxServices
（5）LogHandler（NonAggregatingLogHandler | LogAggregationService）
（6）SharedCacheUploadService

public ContainerManagerImpl(Context context, ContainerExecutor exec,
      DeletionService deletionContext, NodeStatusUpdater nodeStatusUpdater,
      NodeManagerMetrics metrics, ApplicationACLsManager aclsManager,
      LocalDirsHandlerService dirsHandler) {
    super(ContainerManagerImpl.class.getName());
    this.context = context;
    //引入LocalDirsHandlerService检查本地目录健康状况
    this.dirsHandler = dirsHandler;

    // ContainerManager level dispatcher.
    //ContainerManager级别的分发器
    dispatcher = new AsyncDispatcher();
    //引入本地文件删除服务
    this.deletionService = deletionContext;
    //引入NM的metrics统计功能
    this.metrics = metrics;

    //创建资源本地服务
    rsrcLocalizationSrvc =
        createResourceLocalizationService(exec, deletionContext, context);
    addService(rsrcLocalizationSrvc);

    //创建container启动服务
    containersLauncher = createContainersLauncher(context, exec);
    addService(containersLauncher);

    //引入nodeStatusUpdater服务
    this.nodeStatusUpdater = nodeStatusUpdater;
    this.aclsManager = aclsManager;

    // Start configurable services
    auxiliaryServices = new AuxServices();
    auxiliaryServices.registerServiceListener(this);
    addService(auxiliaryServices);

    //创建containerMonitor服务
    this.containersMonitor =
        new ContainersMonitorImpl(exec, dispatcher, this.context);
    addService(this.containersMonitor);

    //dispatcher注册各个子服务
    dispatcher.register(ContainerEventType.class,
        new ContainerEventDispatcher());
    dispatcher.register(ApplicationEventType.class,
        new ApplicationEventDispatcher());
    dispatcher.register(LocalizationEventType.class, rsrcLocalizationSrvc);
    dispatcher.register(AuxServicesEventType.class, auxiliaryServices);
    dispatcher.register(ContainersMonitorEventType.class, containersMonitor);
    dispatcher.register(ContainersLauncherEventType.class, containersLauncher);
    
    addService(dispatcher);

    //创建一个读写锁 TODO：用在哪？
    ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
    this.readLock = lock.readLock();
    this.writeLock = lock.writeLock();
  }

@Override
  public void serviceInit(Configuration conf) throws Exception {
     //创建logHandler服务并注册到分发器
    LogHandler logHandler =
      createLogHandler(conf, this.context, this.deletionService);
    addIfService(logHandler);
    dispatcher.register(LogHandlerEventType.class, logHandler);
    
    // add the shared cache upload service (it will do nothing if the shared
    // cache is disabled)
    //创建共享缓存上传服务并注册到分发器
    SharedCacheUploadService sharedCacheUploader =
        createSharedCacheUploaderService();
    addService(sharedCacheUploader);
    dispatcher.register(SharedCacheUploadEventType.class, sharedCacheUploader);
    
    //container kill等待时间
    waitForContainersOnShutdownMillis =
        conf.getLong(YarnConfiguration.NM_SLEEP_DELAY_BEFORE_SIGKILL_MS,
            YarnConfiguration.DEFAULT_NM_SLEEP_DELAY_BEFORE_SIGKILL_MS) +
        conf.getLong(YarnConfiguration.NM_PROCESS_KILL_WAIT_MS,
            YarnConfiguration.DEFAULT_NM_PROCESS_KILL_WAIT_MS) +
        SHUTDOWN_CLEANUP_SLOP_MS;

    super.serviceInit(conf);
    recover();
  }

资源本地服务ResourceLocalizationService

作用：负责本地初始化，创建相应目录
子服务：LocalizerTracker

@Override
  public void serviceInit(Configuration conf) throws Exception {
    this.validateConf(conf);
    this.publicRsrc = new LocalResourcesTrackerImpl(null, null, dispatcher,
        true, conf, stateStore);
    this.recordFactory = RecordFactoryProvider.getRecordFactory(conf);

    try {
      lfs = getLocalFileContext(conf);
      lfs.setUMask(new FsPermission((short) FsPermission.DEFAULT_UMASK));

      if (!stateStore.canRecover()|| stateStore.isNewlyCreated()) {
        //清理本地数据目录（即删除）
        cleanUpLocalDirs(lfs, delService);
        //创建本地数据目录
        initializeLocalDirs(lfs);
        //创建本地log目录
        initializeLogDirs(lfs);
      }
    } catch (Exception e) {
      throw new YarnRuntimeException(
        "Failed to initialize LocalizationService", e);
    }

    //本地化cache大小 localizer.cache.target-size-mb
    cacheTargetSize =
      conf.getLong(YarnConfiguration.NM_LOCALIZER_CACHE_TARGET_SIZE_MB, YarnConfiguration.DEFAULT_NM_LOCALIZER_CACHE_TARGET_SIZE_MB) << 20;
      //cache清除间隔 localizer.cache.cleanup.interval-ms
    cacheCleanupPeriod =
      conf.getLong(YarnConfiguration.NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS, YarnConfiguration.DEFAULT_NM_LOCALIZER_CACHE_CLEANUP_INTERVAL_MS);
    //NM进程绑定地址，IP：端口
    localizationServerAddress = conf.getSocketAddr(
        YarnConfiguration.NM_BIND_HOST,
        YarnConfiguration.NM_LOCALIZER_ADDRESS,
        YarnConfiguration.DEFAULT_NM_LOCALIZER_ADDRESS,
        YarnConfiguration.DEFAULT_NM_LOCALIZER_PORT);

    //创建LocalizerTracker服务并注册事件 TODO：作用？
    localizerTracker = createLocalizerTracker(conf);
    addService(localizerTracker);
    dispatcher.register(LocalizerEventType.class, localizerTracker);
    super.serviceInit(conf);
  }

相关参数：
yarn.nodemanager.localizer.cache.target-size-mb
yarn.nodemanager.localizer.cache.cleanup.interval-ms
yarn.nodemanager.bind-host NM绑定ip
localizer.address NM地址，默认0.0.0.0:8040

container启动服务ContainersLauncher

作用：负责container启动，初始化过程比较简单

@Override
  protected void serviceInit(Configuration conf) throws Exception {
    try {
      //TODO Is this required?
      FileContext.getLocalFSFileContext(conf);
    } catch (UnsupportedFileSystemException e) {
      throw new YarnRuntimeException("Failed to start ContainersLauncher", e);
    }
    super.serviceInit(conf);
  }

container监控服务ContianerMonitor

作用：container状态监控，主要是内存合理性方面的校验

public ContainersMonitorImpl(ContainerExecutor exec,
      AsyncDispatcher dispatcher, Context context) {
    super("containers-monitor");

    this.containerExecutor = exec;
    this.eventDispatcher = dispatcher;
    this.context = context;

    this.containersToBeAdded = new HashMap<ContainerId, ProcessTreeInfo>();
    this.containersToBeRemoved = new ArrayList<ContainerId>();
    //创建一个线程
    this.monitoringThread = new MonitoringThread();
  }

@Override
  protected void serviceInit(Configuration conf) throws Exception {
    //监控间隔
    this.monitoringInterval =
        conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS,
            YarnConfiguration.DEFAULT_NM_CONTAINER_MON_INTERVAL_MS);

    Class<? extends ResourceCalculatorPlugin> clazz =
        conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR, null,
            ResourceCalculatorPlugin.class);
    //实例化资源计算插件,两个实现：LinuxResourceCalculatorPlugin和WindowsResourceCalculatorPlugin
    this.resourceCalculatorPlugin =
        ResourceCalculatorPlugin.getResourceCalculatorPlugin(clazz, conf);
    LOG.info(" Using ResourceCalculatorPlugin : "
        + this.resourceCalculatorPlugin);
    //进程树类
    processTreeClass = conf.getClass(YarnConfiguration.NM_CONTAINER_MON_PROCESS_TREE, null,
            ResourceCalculatorProcessTree.class);
    this.conf = conf;
    LOG.info(" Using ResourceCalculatorProcessTree : "
        + this.processTreeClass);
    
    //container metrics开启
    this.containerMetricsEnabled =
        conf.getBoolean(YarnConfiguration.NM_CONTAINER_METRICS_ENABLE,
            YarnConfiguration.DEFAULT_NM_CONTAINER_METRICS_ENABLE);
    this.containerMetricsPeriodMs =
        conf.getLong(YarnConfiguration.NM_CONTAINER_METRICS_PERIOD_MS,
            YarnConfiguration.DEFAULT_NM_CONTAINER_METRICS_PERIOD_MS);

    //container的物理内存配置
    long configuredPMemForContainers = conf.getLong(
        YarnConfiguration.NM_PMEM_MB,
        YarnConfiguration.DEFAULT_NM_PMEM_MB) * 1024 * 1024l;

    //container的虚拟核数配置
    long configuredVCoresForContainers = conf.getLong(
        YarnConfiguration.NM_VCORES,
        YarnConfiguration.DEFAULT_NM_VCORES);


    // Setting these irrespective of whether checks are enabled. Required in
    // the UI.
    // / Physical memory configuration //
    this.maxPmemAllottedForContainers = configuredPMemForContainers;
    this.maxVCoresAllottedForContainers = configuredVCoresForContainers;

    // / Virtual memory configuration //
    float vmemRatio = conf.getFloat(YarnConfiguration.NM_VMEM_PMEM_RATIO,
        YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO);
    Preconditions.checkArgument(vmemRatio > 0.99f,
        YarnConfiguration.NM_VMEM_PMEM_RATIO + " should be at least 1.0");
    this.maxVmemAllottedForContainers =
        (long) (vmemRatio * configuredPMemForContainers);

    pmemCheckEnabled = conf.getBoolean(YarnConfiguration.NM_PMEM_CHECK_ENABLED,
        YarnConfiguration.DEFAULT_NM_PMEM_CHECK_ENABLED);
    vmemCheckEnabled = conf.getBoolean(YarnConfiguration.NM_VMEM_CHECK_ENABLED,
        YarnConfiguration.DEFAULT_NM_VMEM_CHECK_ENABLED);
    LOG.info("Physical memory check enabled: " + pmemCheckEnabled);
    LOG.info("Virtual memory check enabled: " + vmemCheckEnabled);

    nodeCpuPercentageForYARN =
        NodeManagerHardwareUtils.getNodeCpuPercentage(conf);

    if (pmemCheckEnabled) {
      // Logging if actual pmem cannot be determined.
      long totalPhysicalMemoryOnNM = UNKNOWN_MEMORY_LIMIT;
      if (this.resourceCalculatorPlugin != null) {
        //从/proc/meminfo文件中读出MemTotal值即为机器总物理内存大小
        totalPhysicalMemoryOnNM = this.resourceCalculatorPlugin
            .getPhysicalMemorySize();
        if (totalPhysicalMemoryOnNM <= 0) {
          LOG.warn("NodeManager's totalPmem could not be calculated. "
              + "Setting it to " + UNKNOWN_MEMORY_LIMIT);
          totalPhysicalMemoryOnNM = UNKNOWN_MEMORY_LIMIT;
        }
      }

      //最大使用内存超过物理内存的80%，则发出警告，因为这很有可能引起OOM
      if (totalPhysicalMemoryOnNM != UNKNOWN_MEMORY_LIMIT &&
          this.maxPmemAllottedForContainers > totalPhysicalMemoryOnNM * 0.80f) {
        LOG.warn("NodeManager configured with "
            + TraditionalBinaryPrefix.long2String(maxPmemAllottedForContainers,
                "", 1)
            + " physical memory allocated to containers, which is more than "
            + "80% of the total physical memory available ("
            + TraditionalBinaryPrefix.long2String(totalPhysicalMemoryOnNM, "",
                1) + "). Thrashing might happen.");
      }
    }
    super.serviceInit(conf);
  }

相关参数：
container-monitor.interval-ms container监控间隔，默认3000ms
container-monitor.resource-calculator.class container资源计算器类，默认ResourceCalculatorPlugin
container-monitor.process-tree.class 进程数类，默认ResourceCalculatorProcessTree
container-metrics.enable 是否开启container metrics统计，默认true
container-metrics.period-ms container metrics flush周期，默认-1在完成是flush
resource.memory-mb container NM为container可分配的内存大小，默认8 * 1024MB
resource.cpu-vcores NM为container可分配的vcore数量，默认8
vmem-pmem-ratio 虚拟内存和物理内存的比率，默认2.1
pmem-check-enabled 是否开启物理内容检查，默认true
vmem-check-enabled 是否开启虚拟内存检查，默认true
resource.percentage-physical-cpu-limit 为container分配的物理cpu占比，默认100%

AuxServices

public class AuxServices extends AbstractService
    implements ServiceStateChangeListener, EventHandler<AuxServicesEvent> {

  static final String STATE_STORE_ROOT_NAME = "nm-aux-services";

  private static final Log LOG = LogFactory.getLog(AuxServices.class);

  protected final Map<String,AuxiliaryService> serviceMap;
  protected final Map<String,ByteBuffer> serviceMetaData;

  private final Pattern p = Pattern.compile("^[A-Za-z_]+[A-Za-z0-9_]*$");

  public AuxServices() {
    super(AuxServices.class.getName());
    serviceMap =
      Collections.synchronizedMap(new HashMap<String,AuxiliaryService>());
    serviceMetaData =
      Collections.synchronizedMap(new HashMap<String,ByteBuffer>());
    // Obtain services from configuration in init()
  }

@Override
  public void serviceInit(Configuration conf) throws Exception {
    final FsPermission storeDirPerms = new FsPermission((short)0700);
    Path stateStoreRoot = null;
    FileSystem stateStoreFs = null;
    //NM是否开启container恢复，yarn.nodemanager.recovery.enabled
    boolean recoveryEnabled = conf.getBoolean(
        YarnConfiguration.NM_RECOVERY_ENABLED,
        YarnConfiguration.DEFAULT_NM_RECOVERY_ENABLED);
    //NM保存container状态的根路径，stateStoreRoot=${yarn.nodemanager.recovery.dir}/${nm-aux-services}
    //NM保存container状态的文件系统，本地，stateStoreFs=localFileSystem
    if (recoveryEnabled) {
      stateStoreRoot = new Path(conf.get(YarnConfiguration.NM_RECOVERY_DIR),
          STATE_STORE_ROOT_NAME);
      stateStoreFs = FileSystem.getLocal(conf);
    }
    //NM上运行的附属服务 yarn.nodemanager.aux-services，需要配置成mapreduce_shuffle才能运行mr程序
    Collection<String> auxNames = conf.getStringCollection(
        YarnConfiguration.NM_AUX_SERVICES);
    for (final String sName : auxNames) {
      try {
        Preconditions
            .checkArgument(
                validateAuxServiceName(sName),
                "The ServiceName: " + sName + " set in " +
                YarnConfiguration.NM_AUX_SERVICES +" is invalid." +
                "The valid service name should only contain a-zA-Z0-9_ " +
                "and can not start with numbers");
        //获取配置的附属服务对于类 yarn.nodemanager.aux-services.%s.class，如yarn.nodemanager.aux-services.mapreduce_shuffle.class=org.apache.hadoop.mapred.ShuffleHandler
        Class<? extends AuxiliaryService> sClass = conf.getClass(
              String.format(YarnConfiguration.NM_AUX_SERVICE_FMT, sName), null,
              AuxiliaryService.class);
        if (null == sClass) {
          throw new RuntimeException("No class defined for " + sName);
        }
        //实例化AuxiliaryService对象
        AuxiliaryService s = ReflectionUtils.newInstance(sClass, conf);
        // TODO better use s.getName()?
        if(!sName.equals(s.getName())) {
          LOG.warn("The Auxilurary Service named '"+sName+"' in the "
                  +"configuration is for "+sClass+" which has "
                  +"a name of '"+s.getName()+"'. Because these are "
                  +"not the same tools trying to send ServiceData and read "
                  +"Service Meta Data may have issues unless the refer to "
                  +"the name in the config.");
        }
        //添加到serviceMap
        addService(sName, s);
        //如果启动恢复container功能则创建状态存储路径（恢复功能开启后，在NM重启后会进行恢复container工作）
        if (recoveryEnabled) {
          Path storePath = new Path(stateStoreRoot, sName);
          stateStoreFs.mkdirs(storePath, storeDirPerms);
          s.setRecoveryPath(storePath);
        }
        //AuxiliaryService对象初始化，看看ShuffleHandler.init
        s.init(conf);
      } catch (RuntimeException e) {
        LOG.fatal("Failed to initialize " + sName, e);
        throw e;
      }
    }
    super.serviceInit(conf);
  }

ShuffleHandler的初始化

//ShuffleHandler.java
@Override
  protected void serviceInit(Configuration conf) throws Exception {
    //是否开启shuffle读操作系统缓存，mapreduce.shuffle.manage.os.cache，默认true
    manageOsCache = conf.getBoolean(SHUFFLE_MANAGE_OS_CACHE,
        DEFAULT_SHUFFLE_MANAGE_OS_CACHE);

    //预先读取数据大小，mapreduce.shuffle.readahead.bytes，默认4M
    readaheadLength = conf.getInt(SHUFFLE_READAHEAD_BYTES,
        DEFAULT_SHUFFLE_READAHEAD_BYTES);

    //响应reduce下载数据请求的最大连接数，mapreduce.shuffle.max.connections，默认0代表连接数无限制。注意：该配置是针对每个NM配置的，而非每个作业配置
    maxShuffleConnections = conf.getInt(MAX_SHUFFLE_CONNECTIONS, 
                                        DEFAULT_MAX_SHUFFLE_CONNECTIONS);
    //响应reduce下载数据请求的最大线程数，mapreduce.shuffle.max.threads，默认0大代表为机器cpu和数的2倍。注意：该配置是针对每个NM配置的，而非每个作业配置
    int maxShuffleThreads = conf.getInt(MAX_SHUFFLE_THREADS,
                                        DEFAULT_MAX_SHUFFLE_THREADS);
    if (maxShuffleThreads == 0) {
      maxShuffleThreads = 2 * Runtime.getRuntime().availableProcessors();
    }
    
    //传输数据的buffer大小，mapreduce.shuffle.transfer.buffer.size，默认128k
    shuffleBufferSize = conf.getInt(SHUFFLE_BUFFER_SIZE, 
                                    DEFAULT_SHUFFLE_BUFFER_SIZE);
        
    //mapreduce.shuffle.transferTo.allowed，默认true
    shuffleTransferToAllowed = conf.getBoolean(SHUFFLE_TRANSFERTO_ALLOWED,
         (Shell.WINDOWS)?WINDOWS_DEFAULT_SHUFFLE_TRANSFERTO_ALLOWED:
                         DEFAULT_SHUFFLE_TRANSFERTO_ALLOWED);

    ThreadFactory bossFactory = new ThreadFactoryBuilder()
      .setNameFormat("ShuffleHandler Netty Boss #%d")
      .build();
    ThreadFactory workerFactory = new ThreadFactoryBuilder()
      .setNameFormat("ShuffleHandler Netty Worker #%d")
      .build();
    
    //创建出selector
    selector = new NioServerSocketChannelFactory(
        Executors.newCachedThreadPool(bossFactory),
        Executors.newCachedThreadPool(workerFactory),
        maxShuffleThreads);
    super.serviceInit(new Configuration(conf));
  }

以上主要是shuffle的相关参数初始化，netty服务初始化。
再来看serviceStart()：配置shuffle httpserver，绑定端口并启动server来监听请求。

@Override
  protected void serviceStart() throws Exception {
    Configuration conf = getConfig();
    userRsrc = new ConcurrentHashMap<String,String>();
    secretManager = new JobTokenSecretManager();
    //恢复状态
    recoverState(conf);
    //netty服务的httpserver
    ServerBootstrap bootstrap = new ServerBootstrap(selector);
    try {
      //处理http请求的pipelineFactory，内部创建出Shuffle对象
      pipelineFact = new HttpPipelineFactory(conf);
    } catch (Exception ex) {
      throw new RuntimeException(ex);
    }
    bootstrap.setPipelineFactory(pipelineFact);
    //mapreduce.shuffle.port，默认13562
    port = conf.getInt(SHUFFLE_PORT_CONFIG_KEY, DEFAULT_SHUFFLE_PORT);
    //server绑定监听端口构建出接受请求的channel
    Channel ch = bootstrap.bind(new InetSocketAddress(port));
    //将channel加入到channelgroup中国呢
    accepted.add(ch);
    port = ((InetSocketAddress)ch.getLocalAddress()).getPort();
    //mapreduce.shuffle.port
    conf.set(SHUFFLE_PORT_CONFIG_KEY, Integer.toString(port));
    //为Shuffle对象设置端口
    pipelineFact.SHUFFLE.setPort(port);
    LOG.info(getName() + " listening on port " + port);
    super.serviceStart();

    //mapreduce.shuffle.ssl.file.buffer.size,默认60K
    sslFileBufferSize = conf.getInt(SUFFLE_SSL_FILE_BUFFER_SIZE_KEY,
                                    DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE);
    //apreduce.shuffle.connection-keep-alive.enable，默认false
    connectionKeepAliveEnabled =
        conf.getBoolean(SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED,
          DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED);
    //mapreduce.shuffle.connection-keep-alive.timeout，默认5
    connectionKeepAliveTimeOut =
        Math.max(1, conf.getInt(SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT,
          DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT));
    //mapreduce.shuffle.mapoutput-info.meta.cache.size，默认1000
    mapOutputMetaInfoCacheSize =
        Math.max(1, conf.getInt(SHUFFLE_MAPOUTPUT_META_INFO_CACHE_SIZE,
          DEFAULT_SHUFFLE_MAPOUTPUT_META_INFO_CACHE_SIZE));
  }

recoverState(conf)

//ShuffleHandler.java
private void recoverState(Configuration conf) throws IOException {
    //recoverRoot=${yarn.nodemanager.recovery.dir}/mapreduce_shuffle
    Path recoveryRoot = getRecoveryPath();
    if (recoveryRoot != null) {
      //创建levelDB对象
      startStore(recoveryRoot);
      //jobPattern=job_[0-9]+_[0-9]+
      Pattern jobPattern = Pattern.compile(JobID.JOBID_REGEX);
      LeveldbIterator iter = null;
      try {
        //为DB创建读取数据的迭代器
        iter = new LeveldbIterator(stateDb);
        iter.seek(bytes(JobID.JOB));
        while (iter.hasNext()) {
          Map.Entry<byte[],byte[]> entry = iter.next();
          String key = asString(entry.getKey());
          if (!jobPattern.matcher(key).matches()) {
            break;
          }
          //解析出需要恢复的job，放入成员变量userRsrc中，userRsrc是个map
          recoverJobShuffleInfo(key, entry.getValue());
        }
      } catch (DBException e) {
        throw new IOException("Database error during recovery", e);
      } finally {
        if (iter != null) {
          iter.close();
        }
      }
    }
  }

startStore(recoveryRoot)

private void startStore(Path recoveryRoot) throws IOException {
    Options options = new Options();
    options.createIfMissing(false);
    options.logger(new LevelDBLogger());
    //dbPath=${yarn.nodemanager.recovery.dir}/mapreduce_shuffle/mapreduce_shuffle_state
    Path dbPath = new Path(recoveryRoot, STATE_DB_NAME);
    LOG.info("Using state database at " + dbPath + " for recovery");
    File dbfile = new File(dbPath.toString());
    try {
      //创建出DB实例
      stateDb = JniDBFactory.factory.open(dbfile, options);
    } catch (NativeDB.DBException e) {
      if (e.isNotFound() || e.getMessage().contains(" does not exist ")) {
        LOG.info("Creating state database at " + dbfile);
        options.createIfMissing(true);
        try {
          stateDb = JniDBFactory.factory.open(dbfile, options);
          storeVersion();
        } catch (DBException dbExc) {
          throw new IOException("Unable to create state store", dbExc);
        }
      } else {
        throw e;
      }
    }
    checkVersion();
  }

log聚合服务 LogHandler(LogAggregationService)

在ContainerManagerImpl.serviceInit中进行LogHandler的创建

//ContainerManagerImpl.java
LogHandler logHandler =
      createLogHandler(conf, this.context, this.deletionService);
addIfService(logHandler);
dispatcher.register(LogHandlerEventType.class, logHandler);

//ContainerManagerImpl.java
protected LogHandler createLogHandler(Configuration conf, Context context,
      DeletionService deletionService) {
    //是否开启log聚合功能，yarn.log-aggregation-enable 默认false
    if (conf.getBoolean(YarnConfiguration.LOG_AGGREGATION_ENABLED,
        YarnConfiguration.DEFAULT_LOG_AGGREGATION_ENABLED)) {
      //log聚合service
      return new LogAggregationService(this.dispatcher, context,
          deletionService, dirsHandler);
    } else {
      //非聚合log的handler
      return new NonAggregatingLogHandler(this.dispatcher, deletionService,
                                          dirsHandler,
                                          context.getNMStateStore());
    }
  }

LogAggregationService的创建、初始化和启动

//LogAggregationService.java
public LogAggregationService(Dispatcher dispatcher, Context context,
      DeletionService deletionService, LocalDirsHandlerService dirsHandler) {
    super(LogAggregationService.class.getName());
    this.dispatcher = dispatcher;
    this.context = context;
    this.deletionService = deletionService;
    this.dirsHandler = dirsHandler;
    this.appLogAggregators =
        new ConcurrentHashMap<ApplicationId, AppLogAggregator>();
    //创建线程池
    this.threadPool = Executors.newCachedThreadPool(
        new ThreadFactoryBuilder()
          .setNameFormat("LogAggregationService #%d")
          .build());
  }

  protected void serviceInit(Configuration conf) throws Exception {
    //applog存放路径，yarn.nodemanager.remote-app-log-dir，默认/tmp/logs
    this.remoteRootLogDir =
        new Path(conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR,
            YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
    //applog存放文件后缀名，yarn.nodemanager.remote-app-log-dir-suffix，默认logs
    //log会被存放在NM_REMOTE_APP_LOG_DIR/${user}/NM_REMOTE_APP_LOG_DIR_SUFFIX/${appId}
    this.remoteRootLogDirSuffix =
        conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR_SUFFIX,
            YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR_SUFFIX);

    super.serviceInit(conf);
  }

  @Override
  protected void serviceStart() throws Exception {
    // NodeId is only available during start, the following cannot be moved
    // anywhere else.
    this.nodeId = this.context.getNodeId();
    super.serviceStart();
  }

NonAggregatingLogHandler创建、初始化、启动

//NonAggregatingLogHandler.java
public NonAggregatingLogHandler(Dispatcher dispatcher,
      DeletionService delService, LocalDirsHandlerService dirsHandler,
      NMStateStoreService stateStore) {
    super(NonAggregatingLogHandler.class.getName());
    this.dispatcher = dispatcher;
    this.delService = delService;
    this.dirsHandler = dirsHandler;
    this.stateStore = stateStore;
    this.appOwners = new ConcurrentHashMap<ApplicationId, String>();
  }

  @Override
  protected void serviceInit(Configuration conf) throws Exception {
    // Default 3 hours.
    //container的log保存事件，yarn.nodemanager.log.retain-seconds，默认3小时
    this.deleteDelaySeconds =
        conf.getLong(YarnConfiguration.NM_LOG_RETAIN_SECONDS,
                YarnConfiguration.DEFAULT_NM_LOG_RETAIN_SECONDS);
    //创建ScheduledThreadPoolExecutor对象
    sched = createScheduledThreadPoolExecutor(conf);
    super.serviceInit(conf);
    //log恢复
    recover();
  }

进入createScheduledThreadPoolExecutor(conf)

 ScheduledThreadPoolExecutor createScheduledThreadPoolExecutor(
     Configuration conf) {
   //log删除线程工厂对象
   ThreadFactory tf =
       new ThreadFactoryBuilder().setNameFormat("LogDeleter #%d").build();
   //log删除线程数，yarn.nodemanager.log.deletion-threads-count，默认4
   sched =
       new ScheduledThreadPoolExecutor(conf.getInt(
           YarnConfiguration.NM_LOG_DELETION_THREADS_COUNT,
           YarnConfiguration.DEFAULT_NM_LOG_DELETE_THREAD_COUNT), tf);
   return sched;
 }

进入recover()

/**
 * 启动线程池调度，到时间后删除每个app的log
 * @throws IOException
 */
private void recover() throws IOException {
  if (stateStore.canRecover()) {
    //加载LevelDB存储的LogDeleters/路径中每个app的状态
    RecoveredLogDeleterState state = stateStore.loadLogDeleterState();
    long now = System.currentTimeMillis();
    for (Map.Entry<ApplicationId, LogDeleterProto> entry :
      state.getLogDeleterMap().entrySet()) {
      ApplicationId appId = entry.getKey();
      LogDeleterProto proto = entry.getValue();
      //计算每个app的删除剩余时间
      long deleteDelayMsec = proto.getDeletionTime() - now;
      if (LOG.isDebugEnabled()) {
        LOG.debug("Scheduling deletion of " + appId + " logs in "
            + deleteDelayMsec + " msec");
      }
      //为每个app创建log删除的Runner对象
      LogDeleterRunnable logDeleter =
          new LogDeleterRunnable(proto.getUser(), appId);
      //调度线程到时间了进行app的log删除
      try {
        sched.schedule(logDeleter, deleteDelayMsec, TimeUnit.MILLISECONDS);
      } catch (RejectedExecutionException e) {
        // Handling this event in local thread before starting threads
        // or after calling sched.shutdownNow().
        logDeleter.run();
      }
    }
  }
}

具体的log删除过程就要看NonAggregatingLogHandler L o g D e l e t e r R u n n a b l e . r u n ( ) 了 （ 1 ） A p p l i c a t o n I m p l 发 送 A p p l i c a t i o n E v e n t T y p e . A P P L I C A T I O N L O G H A N D L I N G F I N I S H E D 事 件 通 知 （ 2 ） 调 用 D e l e t i o n S e r v i c e 服 务 真 正 的 删 除 本 地 a p p 的 l o g （ 3 ） l o g 删 除 完 成 后 从 l e v e l d b 中 删 除 每 个 a p p i d 对 应 的 记 录 ， 即 删 除 k e y = L o g D e l e t e r s / LogDeleterRunnable.run()了 （1）ApplicatonImpl发送ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件通知 （2）调用DeletionService服务真正的删除本地app的log （3）log删除完成后从leveldb中删除每个appid对应的记录，即删除key=LogDeleters/ LogDeleterRunnable.run()了（1）ApplicatonImpl发送ApplicationEventType.APPLICATIONL​OGH​ANDLINGF​INISHED事件通知（2）调用DeletionService服务真正的删除本地app的log（3）log删除完成后从leveldb中删除每个appid对应的记录，即删除key=LogDeleters/appID的记录

//NonAggregatingLogHandler.java$LogDeleterRunnable
 @Override
@SuppressWarnings("unchecked")
/**
 * 删除app的本地log
 */
public void run() {
  List<Path> localAppLogDirs = new ArrayList<Path>();
  //获取本地文件系统
  FileContext lfs = getLocalFileContext(getConfig());
  for (String rootLogDir : dirsHandler.getLogDirsForCleanup()) {
    //获取每个app的本地log路径
    Path logDir = new Path(rootLogDir, applicationId.toString());
    try {
      lfs.getFileStatus(logDir);
      localAppLogDirs.add(logDir);
    } catch (UnsupportedFileSystemException ue) {
      LOG.warn("Unsupported file system used for log dir " + logDir, ue);
      continue;
    } catch (IOException ie) {
      continue;
    }
  }

  // Inform the application before the actual delete itself, so that links
  // to logs will no longer be there on NM web-UI.
  // 真正删除log前，先向ApplicatonImpl发送ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件通知
  NonAggregatingLogHandler.this.dispatcher.getEventHandler().handle(
    new ApplicationEvent(this.applicationId,
      ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED));
  // 调用DeletionService服务去真正的删除log（就是删除本地文件的过程）
  if (localAppLogDirs.size() > 0) {
    NonAggregatingLogHandler.this.delService.delete(user, null,
      (Path[]) localAppLogDirs.toArray(new Path[localAppLogDirs.size()]));
  }
  
  //log删除完成后从leveldb中删除每个appid对应的记录
  try {
    NonAggregatingLogHandler.this.stateStore.removeLogDeleter(
        this.applicationId);
  } catch (IOException e) {
    LOG.error("Error removing log deletion state", e);
  }
}

最后看下ApplicationImpl收到ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED事件后的处理

// Transitions from FINISHED state
.addTransition(ApplicationState.FINISHED,
    ApplicationState.FINISHED,
    ApplicationEventType.APPLICATION_LOG_HANDLING_FINISHED,
    new AppLogsAggregatedTransition())

进入AppLogsAggregatedTransition
(1)从NMContext的applications列表中删除本地log已清理完的app
(2)从aclsManagers中删除本地log已清理完的app
(3)删除levelDB中key=ContainerManager/applications/$appID的记录

static class AppLogsAggregatedTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      ApplicationId appId = event.getApplicationID();
      app.context.getApplications().remove(appId);
      app.aclsManager.removeApplication(appId);
      try {
        app.context.getNMStateStore().removeApplication(appId);
      } catch (IOException e) {
        LOG.error("Unable to remove application from state store", e);
      }
    }
  }

WebServer

作用：NM webserver

public WebServer(Context nmContext, ResourceView resourceView,
      ApplicationACLsManager aclsManager,
      LocalDirsHandlerService dirsHandler) {
    super(WebServer.class.getName());
    this.nmContext = nmContext;
    this.nmWebApp = new NMWebApp(resourceView, aclsManager, dirsHandler);
  }

该service无serviceInit方法，初始化工作和启动都放在了serviceStart中，后续分析。

AsyncDispatcher

见YARN源码剖析（二）：RM启动过程

NodeStatusUpdater

作用：

public NodeStatusUpdaterImpl(Context context, Dispatcher dispatcher,
      NodeHealthCheckerService healthChecker, NodeManagerMetrics metrics) {
    super(NodeStatusUpdaterImpl.class.getName());
    this.healthChecker = healthChecker;
    this.context = context;
    this.dispatcher = dispatcher;
    this.metrics = metrics;
    this.recentlyStoppedContainers =
        new LinkedHashMap<ContainerId, Long>();
    this.pendingCompletedContainers =
        new HashMap<ContainerId, ContainerStatus>();
  }

@Override
  protected void serviceInit(Configuration conf) throws Exception {
    //获得内存和虚拟CPU核心相关配置
    int memoryMb = 
        conf.getInt(
            YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB);
    float vMemToPMem =             
        conf.getFloat(
            YarnConfiguration.NM_VMEM_PMEM_RATIO, 
            YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO); 
    int virtualMemoryMb = (int)Math.ceil(memoryMb * vMemToPMem);
    
    int virtualCores =
        conf.getInt(
            YarnConfiguration.NM_VCORES, YarnConfiguration.DEFAULT_NM_VCORES);
    //创建Resource对象，记录上述资源配置
    this.totalResource = Resource.newInstance(memoryMb, virtualCores);
    //将资源记录至监控系统
    metrics.addResource(totalResource);
    this.tokenKeepAliveEnabled = isTokenKeepAliveEnabled(conf);
    this.tokenRemovalDelayMs =
        conf.getInt(YarnConfiguration.RM_NM_EXPIRY_INTERVAL_MS,
            YarnConfiguration.DEFAULT_RM_NM_EXPIRY_INTERVAL_MS);

    this.minimumResourceManagerVersion = conf.get(
        YarnConfiguration.NM_RESOURCEMANAGER_MINIMUM_VERSION,
        YarnConfiguration.DEFAULT_NM_RESOURCEMANAGER_MINIMUM_VERSION);
    
    // Default duration to track stopped containers on nodemanager is 10Min.
    // This should not be assigned very large value as it will remember all the
    // containers stopped during that time.
    //容器执行完毕的检测时间间隔，默认10分钟
    durationToTrackStoppedContainers =
        conf.getLong(YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS,
          600000);
    if (durationToTrackStoppedContainers < 0) {
      String message = "Invalid configuration for "
        + YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS + " default "
          + "value is 10Min(600000).";
      LOG.error(message);
      throw new YarnException(message);
    }
    if (LOG.isDebugEnabled()) {
      LOG.debug(YARN_NODEMANAGER_DURATION_TO_TRACK_STOPPED_CONTAINERS + " :"
        + durationToTrackStoppedContainers);
    }
    super.serviceInit(conf);
    LOG.info("Initialized nodemanager for " + nodeId + ":" +
        " physical-memory=" + memoryMb + " virtual-memory=" + virtualMemoryMb +
        " virtual-cores=" + virtualCores);
  }

相关参数：
duration-to-track-stopped-containers container执行完毕的检测时间间隔，默认10分钟

NMStateStoreService

NodeManager在serviceInit方法中进行NMStateStoreService的初始化

//NodeManager.java
initAndStartRecoveryStore(conf);

//NodeManager.java
private void initAndStartRecoveryStore(Configuration conf)
      throws IOException {
    //是否开启app恢复，yarn.nodemanager.recovery.enabled 默认false
    boolean recoveryEnabled = conf.getBoolean(
        YarnConfiguration.NM_RECOVERY_ENABLED,
        YarnConfiguration.DEFAULT_NM_RECOVERY_ENABLED);
    if (recoveryEnabled) {
      //获取本地文件系统，file:///
      FileSystem recoveryFs = FileSystem.getLocal(conf);
      //恢复目录，yarn.nodemanager.recovery.dir
      String recoveryDirName = conf.get(YarnConfiguration.NM_RECOVERY_DIR);
      if (recoveryDirName == null) {
        throw new IllegalArgumentException("Recovery is enabled but " +
            YarnConfiguration.NM_RECOVERY_DIR + " is not set.");
      }
      Path recoveryRoot = new Path(recoveryDirName);
      //创建出恢复目录
      recoveryFs.mkdirs(recoveryRoot, new FsPermission((short)0700));
      //创建LevelDB存储服务
      nmStore = new NMLeveldbStateStoreService();
    } else {
      nmStore = new NMNullStateStoreService();
    }
    nmStore.init(conf);
    nmStore.start();
  }

备注：NM的app状态存储服务NMStateStoreService有三种实现：NMLeveldbStateStoreService、NMNullStateStoreService、NMMemoryStateStoreService，实际使用的只有NMLeveldbStateStoreService，其他两个只是用来支持单测的。

nmStore.init(conf);会调到父类的serviceInit

//NMStateStoreService.java
/** Initialize the state storage */
@Override
public void serviceInit(Configuration conf) throws IOException {
  initStorage(conf);
}

再调到子类

//NMLeveldbStateStoreService.java
protected void initStorage(Configuration conf)
   throws IOException {
  //创建出${yarn.nodemanager.recovery.dir}/yarn-nm-state目录，其中yarn-nm-state为DB_NAME
 Path storeRoot = createStorageDir(conf);
 Options options = new Options();
 options.createIfMissing(false);
 options.logger(new LeveldbLogger());
 LOG.info("Using state database at " + storeRoot + " for recovery");
 File dbfile = new File(storeRoot.toString());
 try {
   //通过leveldbjni三方库创建出DB对象
   db = JniDBFactory.factory.open(dbfile, options);
 } catch (NativeDB.DBException e) {
   if (e.isNotFound() || e.getMessage().contains(" does not exist ")) {
     LOG.info("Creating state database at " + dbfile);
     isNewlyCreated = true;
     options.createIfMissing(true);
     try {
       db = JniDBFactory.factory.open(dbfile, options);
       // store version
       storeVersion();
     } catch (DBException dbErr) {
       throw new IOException(dbErr.getMessage(), dbErr);
     }
   } else {
     throw e;
   }
 }
 checkVersion();
}

接下来看nmStore.start();
调到父类的serviceStart

/** Start the state storage for use */
@Override
public void serviceStart() throws IOException {
  startStorage();
}

再调到子类

@Override
protected void startStorage() throws IOException {
}

发现是个空方法，这其实是对leveldb的功能的一个展现：leveldb其实不是一个数据库服务，而仅仅是一个kv存储系统，在使用leveldb的时候外部程序继承进它的库就可以使用它的存储功能了。

Jerry Shao
关注
  • 0
    点赞
  • 踩
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
专栏目录
Yarn源码剖析（一） --- RM与NM服务启动以及心跳通信
weixin_42642341的博客
08-05 2769
前言 在Yarn源码剖析（零） --- spark任务提交到yarn的流程中介绍了Yarn任务提交的流程，按照此篇的阅读指导，该篇文章将会介绍Yarn中ResourceManager、NodeManager的启动，以及两者之间的心跳通信。 Yarn的启动 1. 对于任务的yarn服务的启动当然要从它的启动脚本start-yarn.sh中进行分析，可以看到分别执行了yarn-daemon.sh...
Yarn源码剖析（三）--- ApplicationMaster的启动
weixin_42642341的博客
08-28 7136
前言 在上文Yarn源码剖析（二） --- spark-submit，我们介绍了spark任务通过spark-submit提交任务至yarn申请资源至启动的全流程，本篇将介绍启动过程中ApplicationMaster（后文简称AM）是如何启动。 AM的启动与Container的申请 1. 在Yarn源码剖析（二）中yarnClient最终调用submitApplication方法提交任务，...
/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK 权限不够
大壮的博客
07-26 1659
一、报错问题 IO error: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: 权限不够 根据YARN的异常日志，可以看到提示的信息为“IO error:/var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: Permission denied”。需要登录N...
Hadoop Yarn的磁盘检测
修破立生
12-30 208
背景： 最近在公司的测试服务器上做性能测试， Spark Streaming Job，总是运行一段时间之后就挂掉了。看日志似乎是HDFS出现了问题，数据写不进去了。 但是看了看每台服务器的硬盘，明明都还有空间，最少的都有80%。不管三七二十一，先清一清数据， 把程序启动起来继续跑，并配置了crontab，定时清理磁盘上的日志和临时文件。 当时的预感是YARN上有某种磁盘保护措施，当磁盘使用率达到...
秒懂Yarn：从安装到配置的全流程详解
最新发布
洛秋的博客
07-11 4672
Yarn是一个由Facebook、Google、Exponent 和 Tilde 共同开发的新的包管理工具。它与npm相似，用于管理项目的依赖，但它在性能、安全性和一致性方面做了许多改进。我们详细了解了Yarn的安装、配置及其基本用法。Yarn作为一种新的包管理工具，凭借其快速、安全和一致性的优势，已经成为开发者管理JavaScript依赖包的首选工具之一。希望通过本文的介绍，大家能够快速上手并熟练使用Yarn，提高项目开发和管理的效率。
YARN学习总结-第八节-YARN节点管理
qq_20995587的博客
08-27 2741
NodeManager NodeManager会执行健康检查，可以是检查磁盘或者任意用户指定的检查脚本，如果检查失败，则标记此节点为unhealthy并且报告给ResourceManager，然后停止分配容器到此节点。 磁盘检查 磁盘检查两个目录（local-dirs和log-dirs）分别由yarn.nodemanager.local-dirs和yarn.nodemanager.log-d...
IO error: /var/lib/hadoop-yarn/yarn-nm-recovery/yarn-nm-state/LOCK: 权限不够
qq_33286695的博客
09-09 1575
Cloudera Manager启动Yarn阿报错错误日志如下：解决方案：login the related node server, check the permission and ownership of those directories /var/lib/hadoop-*, I got the following infothey are not right. so we need ...
Yarn源码剖析（四）-- AM的注册与资源调度申请Container及启动
weixin_42642341的博客
09-05 2336
AM注册到RM 1. 从Yarn源码剖析（三）-- ApplicationMaster的启动可知提交应用程序至yarn时最后启动了ApplicationMaster类，所以我们直接来看这个类（是spark自己封装的AM）的main方法，可以看到spark是通过调用AMRMClient客户端来调用相关API来实现AM注册的，以及资源的调度。 amClient = AMRMClient.crea...
Yarn源码剖析（二） --- spark-submit
weixin_42642341的博客
08-10 2030
spark-submit 前言 上文Yarn源码剖析（一） --- RM与NM服务启动以及心跳通信介绍了yarn是如何启动的，本文将介绍在yarn正常启动后，任务是如何通过spark-submit提交到yarn上的。 spark-submit脚本 1. 先来观察一下任务提交时的spark-submit脚本中各个参数的含义（并没列举所有，只列举了关键的几个参数） /spark/bin/s...
flink on yarn-per job源码解析、flink on k8s介绍
weixin_51981189的博客
03-28 1091
YARN per job模式下用户程序在Client端被执行，Client端即执行flink shell命令的执行节点。Client端主要工作就是将用户写的代码转换为JobGraph，向YARN提交应用以执行JobGraph。PipelineExecutor（YarnJobClusterExecutor）将StreamGraph转换为JobGraphClusterDescriptor（YarnClusterDescriptor）通过YARN部署flink集群以执行JobGraph。
yarn日常维护之nm健康状态为false的原因查找
Chain的博客
07-19 1079
最近几天使用yarn集群来布flink，结果发现每次nodemanager只能启动一个，而另一个无法启动，然后取8042端口查看情况，发现NodeHealthyStatus的状态变成了False， 正常情况下应该是true，然后查看日志发现， 2019-07-18 21:45:50,504 WARN org.apache.hadoop.yarn.server.nodemanager.Di...
mapreduce任务提交后不报错，卡在Running Job不执行
qqCEM的博客
08-06 882
1.可能是磁盘容量占用过高 解决方法: 1.硬盘扩容，参考 https://blog.csdn.net/u013431916/article/details/80548069 2.配置yarn-site.xml配置文件 <property> <name>yarn.nodemanager.disk-health-checker.max-disk-util...
关于Yarn源码那些事（七）
夜阑听风
05-16 851
上文说到，需要指定Container所在的NM启动其上的Container，我们看看这个方法的内容：/** * Start a list of containers on this NodeManager. */ @Override public StartContainersResponse startContainers( StartContainersRequest req...
【yarn】Yarn container 日志清理
Mrerlou的博客
06-02 3301
今天说一下 的日志清理机制。的日志目录结构如下图所示。 会在所有目录上为同一个应用程序建立相同的目录结构，并采用轮询的调度方式将这些目录分配给不同Container使用。每个Container将输出三类日志：这个配置为 。由于将所有的运行日志保存到本地磁盘上，因此，随着时间的积累，日志必将越来越多。为了避免大量日志“撑爆”磁盘空间，将定期清理日志文件，该功能由组件（当前存在两种实现：和）完成。总起来说，提供了定期删除（由实现）和日志聚集转存（由实现）两种日志清理机制，默认情况下，采用的是定期删除机制。允
Hadoop运维中遇到的问题（持续更新中......）
程序人生
02-05 5795
1. NodeManager不能启动 症状： org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationService         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localiz
YARN源码分析之ApplicationMaster启动流程之NM端
混混
07-12 1130
上一篇介绍了RMAppMaster在RM端的启动流程，本篇继续介绍下在nodemanager端的启动流程。接着上篇的逻辑继续看下node端的startContainers代码:public StartContainersResponse startContainers(StartContainersRequest requests) throws YarnException,
【资源调度总纲】Yarn源码剖析（零） --- spark任务提交到yarn的流程
weixin_42642341的博客
07-31 2058
前言 本系列的目的在于试图剖析spark任务提交至hadoop yarn上的整个过程，从yarn的启动，以及spark-submit提交任务到yarn上，和在yarn中启动任务包括yarn组件之间的通信，用以提升自身知识储备，记录学习的过程为目的，由于个人能力有限文章中或许有理解不到位的地方，敬请谅解。本系列侧重的部分是yarn相关的部分，所以在spark-submit代码剖析阶段会缺少与spa...
Hadoop 2.0 YARN深度剖析：架构与实战详解
2. **源代码解析**：作者从源代码层面出发，剖析YARN架构的实现原理，包括ResourceManager、ResourceScheduler（如FairScheduler和CapacityScheduler）以及NodeManager等关键组件的工作机制。这有助于读者掌握YARN...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值