yarn3.2源码分析之NM端startContainer的流程（二）

概述

NM端startContainer会经历一系列event：

1. request_resource_localization类型的LocalizerEvent 
   LocalizerTracker是LocalizerEvent的事件处理器。LocalizerTracker根据资源的类型进行资源本地化。资源有3种类型：public、private和application。public resources的资源本地化使用PublicLocalizer完成，它通过线程池异步下载资源。private和application的资源本地化将启动独立的进程ContainerLocalizer执行下载资源的工作。
2. localized类型的ResourceEvent 
3. resource_localized类型的ContainerEvent 
4. scheduler_container类型的ContainerSchedulerEvent
5. LaunchContainer类型的ContainersLaunchEvent

LocalizerEvent的处理器 —— LocalizerTracker

处理request_resource_localization类型逻辑如下：

private final PublicLocalizer publicLocalizer;

@Override
    public void handle(LocalizerEvent event) {
      String locId = event.getLocalizerId();
      switch (event.getType()) {
      case REQUEST_RESOURCE_LOCALIZATION:
        // 0) find running localizer or start new thread
        LocalizerResourceRequestEvent req =
          (LocalizerResourceRequestEvent)event;
        switch (req.getVisibility()) {
        case PUBLIC:
//将请求加入publicLocalizer的内部队列，再由线程池执行异步下载资源的任务
          publicLocalizer.addResource(req);
          break;
        case PRIVATE:
        case APPLICATION:
          synchronized (privLocalizers) {
            LocalizerRunner localizer = privLocalizers.get(locId);
            if (localizer != null && localizer.killContainerLocalizer.get()) {
              // Old localizer thread has been stopped, remove it and creates
              // a new localizer thread.
              LOG.info("New " + event.getType() + " localize request for "
                  + locId + ", remove old private localizer.");
              cleanupPrivLocalizers(locId);
              localizer = null;
            }
            if (null == localizer) {
              LOG.info("Created localizer for " + locId);
              localizer = new LocalizerRunner(req.getContext(), locId);
              privLocalizers.put(locId, localizer);
//启动LocalizerRunner线程，LocalizerRunner线程在运行中会启动ContainerLocalizer进程下载资源
              localizer.start();
            }
            // 1) propagate event
            localizer.addResource(req);
          }
          break;
        }
        break;
      }
    }

假设req是public的，调用PublicLocalizer#addResource()方法。

PublicLocalizer介绍

在addResource()方法中，将LocalizerResourceRequestEvent放入线程池中，执行异步下载资源，代表异步执行的Future放入ExecutorCompletionService中存储，该future与LocalizerResourceRequestEvent的对应关系放入pending中存储；

在run()方法中，循环从ExecutorCompletionService中取出一个future，调用future#get()方法阻塞等待该future完成，然后创建ResourceLocalizedEvent.

 class PublicLocalizer extends Thread {

    final FileContext lfs;
    final Configuration conf;
    final ExecutorService threadPool;
    final CompletionService<Path> queue;
    // Its shared between public localizer and dispatcher thread.
//存储future与LocalizerResourceRequestEvent的对应关系
    final Map<Future<Path>,LocalizerResourceRequestEvent> pending;

    PublicLocalizer(Configuration conf) {
      super("Public Localizer");
      this.lfs = getLocalFileContext(conf);
      this.conf = conf;
      this.pending = Collections.synchronizedMap(
          new HashMap<Future<Path>, LocalizerResourceRequestEvent>());
      this.threadPool = createLocalizerExecutor(conf);
      this.queue = new ExecutorCompletionService<Path>(threadPool);
    }

    public void addResource(LocalizerResourceRequestEvent request) {
      // TODO handle failures, cancellation, requests by other containers
      LocalizedResource rsrc = request.getResource();
      LocalResourceRequest key = rsrc.getRequest();
      LOG.info("Downloading public resource: " + key);
      /*
       * Here multiple containers may request the same resource. So we need
       * to start downloading only when
       * 1) ResourceState == DOWNLOADING
       * 2) We are able to acquire non blocking semaphore lock.
       * If not we will skip this resource