Hadoop2.8.5 容器的投运

上一篇我们考察了YARN调度系统的容器周转和分配，RM受理作业后就为该作业分配容器，最后由发射架将容器发送到对岸的NodeManager上，现在我们来看NM收到容器后如何启动JVM并创建AM作为作业的领头人，之后的事情就交给了AM。今天我们就来考察容器投运到NM这一过程。为了投运一个作业,RM 首先得在某个NodeManag er 节点上启动一个进程作为这个作业的主管,扮演类似于项目组长那样的角色,这就是 ApplicationMaster 即 AM ,对于 MapReduce作业就是 MRAppMaster 。后者是前者的一种特例,其 Application 是专门进行 MapReduce 计算的 App 。然而,虽然这个 App 本身是多任务的作业,其 ApplicationMaster 或 MRAppMaster 却只是单任务的作业,所以 ApplicationMaster 或 MRAppMaster 的投运其实只相当于一个单任务的投运。而且, RM 之于 ApplicationMaster 的投运又恰如 ApplicationMaster 之于具体计算任务(如 MapTask )的投运,其作用和过程都很相似。

1. 客户端（RM）投运容器

resourcemanager\amlauncher\ApplicationMasterLauncher.java
AM发射架，发射AM容器

public class ApplicationMasterLauncher extends AbstractService implements EventHandler<AMLauncherEvent> {
  
  // createRunnableLauncher : 创建了AMLauncher任务
  private void launch(RMAppAttempt application) {
    Runnable launcher = createRunnableLauncher(application, 
        AMLauncherEventType.LAUNCH);
    masterEvents.add(launcher);
  }
  
  private class LauncherThread extends Thread {
    @Override
    public void run() {
      while (!this.isInterrupted()) {
        Runnable toLaunch;
        try {
          toLaunch = masterEvents.take();
          launcherPool.execute(toLaunch); //AMLauncher.run()
        } catch (InterruptedException e) {
          return;
        }
      }
    }
  }    
  
}

server\resourcemanager\amlauncher\AMLauncher.java
AM容器发射任务

public class AMLauncher implements Runnable {
  //RPC通讯协议
  private ContainerManagementProtocol containerMgrProxy;
  
  @SuppressWarnings("unchecked")
  public void run() {
    switch (eventType) {
    case LAUNCH:
      try {
        launch();
        //AppAttempt 转入 LAUNCHED,RMAppAttemptImpl 会启动一个 AMLivelinessMonitor 监视 AM 是否存活
        handler.handle(new RMAppAttemptEvent(application.getAppAttemptId(),
            RMAppAttemptEventType.LAUNCHED));
      } catch(Exception ie) {
        ......
      }
      break;
    }
  }
  
  //发射容器
  private void launch() throws IOException, YarnException {
    connect(); //RPC
    ContainerId masterContainerID = masterContainer.getId();
    //ApplicationSubmissionContext 来自用户的作业提交
    ApplicationSubmissionContext applicationContext =application.getSubmissionContext(); 
    // ContainerLaunchContext 用于让 NM 节点创建进程以执行任务
    ContainerLaunchContext launchContext = createAMContainerLaunchContext(applicationContext, masterContainerID);
    //RPC请求
    StartContainerRequest scRequest = StartContainerRequest.newInstance(launchContext,
          masterContainer.getContainerToken());
    List<StartContainerRequest> list = new ArrayList<StartContainerRequest>();
    list.add(scRequest);
    StartContainersRequest allRequests = StartContainersRequest.newInstance(list);
    //RPC启动容器
    StartContainersResponse response = containerMgrProxy.startContainers(allRequests);
    }
  }
  
  private void connect() throws IOException {
    ContainerId masterContainerID = masterContainer.getId();
    containerMgrProxy = getContainerMgrProxy(masterContainerID);
  }

  protected ContainerManagementProtocol getContainerMgrProxy(final ContainerId containerId) {
    //主容器中有NM的 NodeId
    final NodeId node = masterContainer.getNodeId();
    //还有对方的 IP 地址和端口号
    final InetSocketAddress containerManagerConnectAddress =
        NetUtils.createSocketAddrForHost(node.getHost(), node.getPort());
    //创建底层 RPC
    final YarnRPC rpc = getYarnRPC();
    //返回代理，这个时候NM作为SERVER端
    //CLIENT : ContainerManagementProtocolPBClientImpl
    return NMProxy.createNMProxy(conf, ContainerManagementProtocol.class,
        currentUser, rpc, containerManagerConnectAddress);
  }
}

ContainerManagementProtocolPBClientImpl.java
向NM节点发送RPC请求

public class ContainerManagementProtocolPBClientImpl implements ContainerManagementProtocol, Closeable {
  //Client端，启动容器
  public StartContainersResponse startContainers(StartContainersRequest requests) throws IOException {
    //获取请求报文协议
    StartContainersRequestProto requestProto = ((StartContainersRequestPBImpl) requests).getProto();
    try {
      //由代理执行： proxy : StartContainersRequestPBImpl
      return new StartContainersResponsePBImpl(proxy.startContainers(null, requestProto));
    } catch (ServiceException e) {
      return null;
    }
  }
}

2. 服务端（NM）启动容器

ContainerManagementProtocolPBServiceImpl.java
接收来自客户端的RPC请求

public class ContainerManagementProtocolPBServiceImpl implements ContainerManagementProtocolPB  {
  @Override
  public StartContainersResponseProto startContainers(RpcController arg0,
      StartContainersRequestProto proto) throws ServiceException {
    StartContainersRequestPBImpl request = new StartContainersRequestPBImpl(proto);
    try {
      //real 实为 ContainerManagerImpl
      StartContainersResponse response = real.startContainers(request);
      return ((StartContainersResponsePBImpl)response).getProto();
    } catch (YarnException e) {
      throw new ServiceException(e);
    } catch (IOException e) {
      throw new ServiceException(e);
    }
  }
}

ContainerManagerImpl.java

服务端 ContainerManagementProtocolPBServiceImpl 将 startContainers转发给 ContainerManagerImpl

public class ContainerManagerImpl extends CompositeService implements ContainerManagementProtocol,
    EventHandler<ContainerManagerEvent> {
    //启动容器
    public StartContainersResponse startContainers(
      StartContainersRequest requests) throws YarnException, IOException {
    synchronized (this.context) {
      for (StartContainerRequest request : requests
          .getStartContainerRequests()) {
        ContainerId containerId = null;
        try {
          //处理请求
          startContainerInternal(nmTokenIdentifier, containerTokenIdentifier,
              request);
          succeededContainers.add(containerId);
        } catch (YarnException e) {
        }
      }
      return StartContainersResponse
          .newInstance(getAuxServiceMetaData(), succeededContainers,
              failedContainers);
    }
  }
  
  //初始化容器
  private void startContainerInternal(NMTokenIdentifier nmTokenIdentifier,
      ContainerTokenIdentifier containerTokenIdentifier,
      StartContainerRequest request) throws YarnException, IOException {
    //容器ID
    ContainerId containerId = containerTokenIdentifier.getContainerID();
    String containerIdStr = containerId.toString();
    String user = containerTokenIdentifier.getApplicationSubmitter();
    //从 request 中恢复出 ContainerLaunchContext ,即 CLC
    ContainerLaunchContext launchContext = request.getContainerLaunchContext();
    //在 NM 节点上创建一个 ContainerImpl 对象
    Container container = new ContainerImpl(getConfig(), this.dispatcher,
            launchContext, credentials, metrics, containerTokenIdentifier, context);
    this.readLock.lock();
    try {
      if (!serviceStopped) {
        // 创建一个 ApplicationImpl注意这个 dispatcher 就是 ContainerManagerImpl.dispatcher
        Application application =
            new ApplicationImpl(dispatcher, user, applicationID, credentials, context);
        if (null == context.getApplications().putIfAbsent(applicationID, application)) {
          context.getNMStateStore().storeApplication(applicationID,
              buildAppProto(applicationID, user, credentials, appAcls,
                logAggregationContext));
          //产生一个 INIT _ APPLICATION 事件，用来触发这个 ApplicationImpl 的状态机
          dispatcher.getEventHandler().handle(
            new ApplicationInitEvent(applicationID, appAcls,
              logAggregationContext));
        }

        this.context.getNMStateStore().storeContainer(containerId,
            containerTokenIdentifier.getVersion(), request);
         //用来运载这个新建的 Container，产生一个 ApplicationEventType.INIT _ CONTAINER 事件
         //把 Container 交给 ApplicationImpl
        dispatcher.getEventHandler().handle(
          new ApplicationContainerInitEvent(container));
        //用于统计目的
        metrics.launchedContainer();
        metrics.allocateContainer(containerTokenIdentifier.getResource());
      }
    } finally {
      this.readLock.unlock();
    }
  }
}

server\nodemanager\containermanager\application\ApplicationImpl.java

将收到的容器交给ApplicationImpl 处理

public class ApplicationImpl implements Application {
    //跳变规则
	addTransition ( ApplicationState.NEW , ApplicationState.INITING ,
                    ApplicationEventType.INIT _ APPLICATION , newAppInitTransition ())
                    
    static class AppInitTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      ApplicationInitEvent initEvent = (ApplicationInitEvent)event;
      app.applicationACLs = initEvent.getApplicationACLs();
      app.aclsManager.addApplication(app.getAppId(), app.applicationACLs);
      // 触发日志服务的状态机,启动具体 App 的运行日志( LOG )机制
      app.logAggregationContext = initEvent.getLogAggregationContext();
      app.dispatcher.getEventHandler().handle(
          new LogHandlerAppStartedEvent(app.appId, app.user,
              app.credentials, app.applicationACLs,
              app.logAggregationContext));
    }
  }

  ddTransition ( ApplicationState.INITING , ApplicationState.INITING ,
				 ApplicationEventType.INIT _ CONTAINER ,newInitContainerTransition ())
				 
  static class InitContainerTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      ApplicationContainerInitEvent initEvent =
        (ApplicationContainerInitEvent) event;
      Container container = initEvent.getContainer();
      //将 container 放在这个 ApplicationImpl 的 containers 集合中
      app.containers.put(container.getContainerId(), container);
      ......
    }
  }
  
  //启动资源定位服务
  static class AppLogInitDoneTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      app.dispatcher.getEventHandler().handle(
          new ApplicationLocalizationEvent(
              LocalizationEventType.INIT_APPLICATION_RESOURCES, app));
    }
  }
}

server\nodemanager\containermanager\localizer\ResourceLocalizationService.java

资源定位服务，处理本地资源跟踪

public class ResourceLocalizationService extends CompositeService
    implements EventHandler<LocalizationEvent>, LocalizationProtocol {
  //处理事务
  public void handle(LocalizationEvent event) {
    // TODO: create log dir as $logdir/$user/$appId
    switch (event.getType()) {
    case INIT_APPLICATION_RESOURCES:
      handleInitApplicationResources(
          ((ApplicationLocalizationEvent)event).getApplication());
      break;
    case INIT_CONTAINER_RESOURCES:
      handleInitContainerResources((ContainerLocalizationRequestEvent) event);
      break;
    case CONTAINER_RESOURCES_LOCALIZED:
      handleContainerResourcesLocalized((ContainerLocalizationEvent) event);
      break;
    case CACHE_CLEANUP:
      handleCacheCleanup();
      break;
    case CLEANUP_CONTAINER_RESOURCES:
      handleCleanupContainerResources((ContainerLocalizationCleanupEvent)event);
      break;
    case DESTROY_APPLICATION_RESOURCES:
      handleDestroyApplicationResources(
          ((ApplicationLocalizationEvent)event).getApplication());
      break;
    default:
      throw new YarnRuntimeException("Unknown localization event: " + event);
    }
  }
  
  //LocalResourcesTrackerImpl 构造函数的第四个参数是 useLocalCacheDirectoryManager ,意为
  //是否使用 LocalCacheDirectoryManager ,即本地的缓存目录管理
  @SuppressWarnings("unchecked")
  private void handleInitApplicationResources(Application app) {
    String userName = app.getUser();
    //创建一个 LocalResourcesTrackerImpl 对象,并将其放在 privateRsrc 集合中
    privateRsrc.putIfAbsent(userName, new LocalResourcesTrackerImpl(userName,
        null, dispatcher, true, super.getConfig(), stateStore, dirsHandler));
    String appIdStr = app.getAppId().toString();
    //再创建一个 LocalResourcesTrackerImpl 对象,并将其放在 appRsrc 集合中
    appRsrc.putIfAbsent(appIdStr, new LocalResourcesTrackerImpl(app.getUser(),
        app.getAppId(), dispatcher, false, super.getConfig(), stateStore,
        dirsHandler));
     //实际上是 ApplicationEventType.APPLICATION _ INITED
    dispatcher.getEventHandler().handle(new ApplicationInitedEvent(
          app.getAppId()));
  }
}

public class ApplicationImpl implements Application {
   //跳变规则
   addTransition ( ApplicationState.INITING , ApplicationState.RUNNING ,
                   ApplicationEventType.APPLICATION _ INITED , newAppInitDoneTransition ())
  @SuppressWarnings("unchecked")
   static class AppInitDoneTransition implements
      SingleArcTransition<ApplicationImpl, ApplicationEvent> {
    @Override
    public void transition(ApplicationImpl app, ApplicationEvent event) {
      // 对于 containers 集合中的每个容器， 产生一个 INIT _ CONTAINER 事件
      for (Container container : app.containers.values()) {
        app.dispatcher.getEventHandler().handle(new ContainerInitEvent(
              container.getContainerId()));
      }
    }
  }
}

server\nodemanager\containermanager\container\ContainerImpl.java
容器初始化，本地化资源，本地资源分共有，非共有两部分

public class ContainerImpl implements Container {
   //跳变规则
  addTransition(ContainerState.NEW, EnumSet.of(ContainerState.LOCALIZING, ContainerState.LOCALIZED,
                 ContainerState.LOCALIZATION_FAILED, ContainerState.DONE),
        ContainerEventType.INIT_CONTAINER, new RequestResourcesTransition())
  @SuppressWarnings("unchecked") // dispatcher not typed
  static class RequestResourcesTransition implements
      MultipleArcTransition<ContainerImpl,ContainerEvent,ContainerState> {
    @Override
    public ContainerState transition(ContainerImpl container,
        ContainerEvent event) {
      //容器的恢复已经完成(以前保存了容器内容,但是执行失败,需要重执)
      if (container.recoveredStatus == RecoveredContainerStatus.COMPLETED) {
        //发送 ApplicationContainerFinishedEvent 等事件
        container.sendFinishedEvents();
        return ContainerState.DONE;
      } else if (container.recoveredAsKilled &&
          container.recoveredStatus == RecoveredContainerStatus.REQUESTED) {
        // container was killed but never launched
        container.metrics.killedContainer();
        container.metrics.releaseContainer(container.resource);
        container.sendFinishedEvents(); //资源本地化已完成
        return ContainerState.DONE;
      }
      //这是容器发起上下文 CLC
      final ContainerLaunchContext ctxt = container.launchContext;
      //为统计目的
      container.metrics.initingContainer();
      container.dispatcher.getEventHandler().handle(new AuxServicesEvent
          (AuxServicesEventType.CONTAINER_INIT, container));

      container.containerLocalizationStartTime = clock.getTime();
      // 发送资源请求
      Map<String,LocalResource> cntrRsrc = ctxt.getLocalResources();//从 CLC 获取本地资源
      if (!cntrRsrc.isEmpty()) { //有资源请求
        try {
          for (Map.Entry<String,LocalResource> rsrc : cntrRsrc.entrySet()) {//对于其中的每一项资源
            try {
              //创建 LocalResourceRequest 对象
              LocalResourceRequest req = new LocalResourceRequest(rsrc.getValue());
              List<String> links = container.pendingResources.get(req);
              if (links == null) { //如果不在 pendingResources 集合中
                links = new ArrayList<String>();
                //放入这个集合
                container.pendingResources.put(req, links);
              }
              links.add(rsrc.getKey());
              switch (rsrc.getValue().getVisibility()) {
              case PUBLIC: //这是公共资源
                container.publicRsrcs.add(req);
                break;
              case PRIVATE: //这是本容器的专用资源
                container.privateRsrcs.add(req);
                break;
              case APPLICATION: //这是由本应用所属容器共享的 App 资源
                container.appRsrcs.add(req);
                break;
              }
            } 
          }
        } 
        Map<LocalResourceVisibility, Collection<LocalResourceRequest>> req =
            new LinkedHashMap<LocalResourceVisibility,
                        Collection<LocalResourceRequest>>();
        //填写公共资源请求
        if (!container.publicRsrcs.isEmpty()) {
          req.put(LocalResourceVisibility.PUBLIC, container.publicRsrcs);
        }
        //填写专用资源请求
        if (!container.privateRsrcs.isEmpty()) {
          req.put(LocalResourceVisibility.PRIVATE, container.privateRsrcs);
        }
        //填写 App 资源请求
        if (!container.appRsrcs.isEmpty()) {
          req.put(LocalResourceVisibility.APPLICATION, container.appRsrcs);
        }
        //将资源本地化请求作为一个事件发送给 ResourceLocalizationService
        container.dispatcher.getEventHandler().handle(
              new ContainerLocalizationRequestEvent(container, req));
        //ContainerImpl 转入 LOCALIZING 状态
        return ContainerState.LOCALIZING;
      }
    }
  }
}

对于CLC 即 ContainerLaunchContext 中的资源要求,这里先进行了一番分类和整理,放在按资源开放范围即 Visibility 分类( PUBLIC 、 PRIVATE 、 APPLICATION )的 MAP 中。然后将此MAP 搭载在一个 LocalizationEventType. INIT _ CONTAINER _ RESOURCES 事件上发送出去。登记接受 LocalizationEventType 类事件的是一个 ResourceLocalizationService 对象,负责提供资源本地化服务。这个事件说是 INIT _ CONTAINER _ RESOURCES ,其实就是进行资源本地化的请求。资源的本地化当然会有个过程,现在才刚开始,所以这个具体容器的状态变成 LOCALIZING ,表示正在进行之中.

public class ResourceLocalizationService {

  public void handle(LocalizationEvent event) {
   case INIT_CONTAINER_RESOURCES:
       handleInitContainerResources((ContainerLocalizationRequestEvent) event);
       break;
  }

  private void handleInitContainerResources(ContainerLocalizationRequestEvent rsrcReqs) {
    //从该事件对象中获取所涉及的容器
    Container c = rsrcReqs.getContainer();
    //建立一个缓存
    LoadingCache<Path,Future<FileStatus>> statCache =
        CacheBuilder.newBuilder().build(FSDownload.createStatusCacheLoader(getConfig()));
    //从容器中获取相关信息,连同缓存一起创建一个本地化上下文 LocalizerContext
    LocalizerContext ctxt = new LocalizerContext(
        c.getUser(), c.getContainerId(), c.getCredentials(), statCache);
    //将搭载在事件上的资源请求转移到一个 Map 中
    Map<LocalResourceVisibility, Collection<LocalResourceRequest>> rsrcs =
      rsrcReqs.getRequestedResources();
    for (Map.Entry<LocalResourceVisibility, Collection<LocalResourceRequest>> e :
         rsrcs.entrySet()) {
      //根据资源的开放类别,即 e.getKey (),找到相应的本地资源跟踪者
      LocalResourcesTracker tracker =
          getLocalResourcesTracker(e.getKey(), c.getUser(),
              c.getContainerId().getApplicationAttemptId()
                  .getApplicationId());
      for (LocalResourceRequest req : e.getValue()) { //对于相同可见性的每一项资源:
        //LocalResourcesTrackerImpl处理
        tracker.handle(new ResourceRequestEvent(req, e.getKey(), ctxt));
        }
      }
    }
  }

}

server\nodemanager\containermanager\localizer\LocalResourcesTrackerImpl.java
资源跟踪定位

class LocalResourcesTrackerImpl implements LocalResourcesTracker {
 @Override
  public synchronized void handle(ResourceEvent event) {
    //从事件中抽取资源请求
    LocalResourceRequest req = event.getLocalResourceRequest();
    LocalizedResource rsrc = localrsrc.get(req);
    //根据该项资源的可见性找到相应的 LocalizedResource 对象
    switch (event.getType()) {
    case REQUEST:
      if (null == rsrc) { //尚未创建 LocalizedResource
        rsrc = new LocalizedResource(req, dispatcher);
        localrsrc.put(req, rsrc);
      }
      break;
     //向 LocalizedResource 转交该事件
     rsrc.handle(event);

    }
  }

}

3.客户端资源本地化

恢复资源请求，并转发请求给 ResourceLocalizationService

public class LocalizedResource implements EventHandler<ResourceEvent> {
   //跳转规则
   addTransition ( ResourceState.INIT , ResourceState.DOWNLOADING ,
                   ResourceEventType.REQUEST , newFetchResourceTransition ())
    
  @SuppressWarnings("unchecked") // dispatcher not typed
  private static class FetchResourceTransition extends ResourceTransition {
    @Override
    public void transition(LocalizedResource rsrc, ResourceEvent event) {
      //将事件的类型投射恢复成 ResourceRequestEvent
      ResourceRequestEvent req = (ResourceRequestEvent) event;
      //从事件对象中恢复 LocalizerContext
      LocalizerContext ctxt = req.getContext();
      //从 LocalizerContext 中恢复容器
      ContainerId container = ctxt.getContainerId();
      rsrc.ref.add(container);
      //另创一个 REQUEST _ RESOURCE _ LOCALIZATION 事件
      //用这个事件向 ResourceLocalizationService 提出请求
      rsrc.dispatcher.getEventHandler().handle(
          new LocalizerResourceRequestEvent(rsrc, req.getVisibility(), ctxt, 
              req.getLocalResourceRequest().getPattern()));
    }
  }               
}

ResourceLocalizationService.java
资源定位服务开始处理共有/非共有资源

public class ResourceLocalizationService {

    @Override
    public void handle(LocalizerEvent event) {
      //每项具体的资源都有个 LocalizerId
      String locId = event.getLocalizerId();
      switch (event.getType()) {
      case REQUEST_RESOURCE_LOCALIZATION:
        // 0) find running localizer or start new thread
        LocalizerResourceRequestEvent req = (LocalizerResourceRequestEvent)event;
        switch (req.getVisibility()) {
        case PUBLIC:
          ///公共资源的本地化交由 publicLocalizer 线程完成
          publicLocalizer.addResource(req);
          break;
        case PRIVATE:
        case APPLICATION:
          synchronized (privLocalizers) {
            //获取非公共资源本地化线程
            LocalizerRunner localizer = privLocalizers.get(locId);
            if (null == localizer) {
              //创建非公共资源本地化线程
              localizer = new LocalizerRunner(req.getContext(), locId);
              privLocalizers.put(locId, localizer);
              localizer.start();//启动这个线程
            }
            //交给非公共资源本地化线程去完成
            localizer.addResource(req);
          }
          break;
        }
        break;
      }
    }
}

ResourceLocalizationService.java
公共资源处理线程

class PublicLocalizer extends Thread {
    final FileContext lfs;
    final Configuration conf;
    final ExecutorService threadPool;//线程池
    final CompletionService<Path> queue;//等待被线程池中的线程加以执行的队列
    // Its shared between public localizer and dispatcher thread.
    final Map<Future<Path>,LocalizerResourceRequestEvent> pending;

    PublicLocalizer(Configuration conf) {
      super("Public Localizer");
      this.lfs = getLocalFileContext(conf);
      this.conf = conf;
      this.pending = Collections.synchronizedMap(
          new HashMap<Future<Path>, LocalizerResourceRequestEvent>());
      this.threadPool = createLocalizerExecutor(conf);
      this.queue = new ExecutorCompletionService<Path>(threadPool);
    }

    public void addResource(LocalizerResourceRequestEvent request) {
      // TODO handle failures, cancellation, requests by other containers
      LocalizedResource rsrc = request.getResource();
      LocalResourceRequest key = rsrc.getRequest();
      if (rsrc.tryAcquire()) {
        //下载中
        if (rsrc.getState() == ResourceState.DOWNLOADING) {
          //本地资源请求
          LocalResource resource = request.getResource().getRequest();
          try {
            //缓存路径
            Path publicRootPath =
                dirsHandler.getLocalPathForWrite("." + Path.SEPARATOR
                    + ContainerLocalizer.FILECACHE,
                  ContainerLocalizer.getEstimatedSize(resource), true);
            //目标文件路径
            Path publicDirDestPath =
                publicRsrc.getPathForLocalization(key, publicRootPath,
                    delService);
            //挂起提交一项FSDownload任务
            synchronized (pending) {
              pending.put(queue.submit(new FSDownload(lfs, null, conf,
                  publicDirDestPath, resource, request.getContext().getStatCache())),
                  request);
            }
          } catch (IOException e) {
            ......
          }
        } else {
          rsrc.unlock();
        }
      }
    }

    @Override
    public void run() {
      try {
        // TODO shutdown, better error handling esp. DU
        while (!Thread.currentThread().isInterrupted()) {
          try {
            //从队列中取任务
            Future<Path> completed = queue.take();
            LocalizerResourceRequestEvent assoc = pending.remove(completed);
            try {
              Path local = completed.get();
              LocalResourceRequest key = assoc.getResource().getRequest();
              //用 LOCALIZED 事件驱动 LocalizedResource 的状态机
              publicRsrc.handle(new ResourceLocalizedEvent(key, local, FileUtil
                .getDU(new File(local.toUri()))));
              assoc.getResource().unlock();
            }
          } catch (InterruptedException e) {
            return;
          }
        }
      }
    }

  }

hadoop-yarn-common\src\main\java\org\apache\hadoop\yarn\util\FSDownload.java

定位到资源后由FSDownload下载到本地

public class FSDownload implements Callable<Path> {
  @Override
  public Path call() throws Exception {
    final Path sCopy;
    //获取该项公共资源的源路径
    //资源描述 resource 所提供的是一种 YarnURL ,需要将其转换成 Hadoop 的 URI
    sCopy = resource.getResource().toPath();
    //创建目标目录
    createDir(destDirPath, cachePerms);
    //加后缀_ tmp 作为工作目录名
    final Path dst_work = new Path(destDirPath + "_tmp");
    //创建工作目录
    createDir(dst_work, cachePerms);
    //目标文件名
    Path dFinal = files.makeQualified(new Path(dst_work, sCopy.getName()));
    try {
      //copy ： 复制文件,可以跨节点、跨文件系统复制
      Path dTmp = null == userUgi ? files.makeQualified(copy(sCopy, dst_work))
          : userUgi.doAs(new PrivilegedExceptionAction<Path>() {
            public Path run() throws Exception {
              return files.makeQualified(copy(sCopy, dst_work));
            };
          });
      unpack(new File(dTmp.toUri()), new File(dFinal.toUri()));
      changePermissions(dFinal.getFileSystem(conf), dFinal);
      //更改工作文件名
      files.rename(dst_work, destDirPath, Rename.OVERWRITE);
    } 
    //返回本地的目标目录路径 + 文件名
    return files.makeQualified(new Path(destDirPath, sCopy.getName()));
  }
}

server\nodemanager\containermanager\localizer\LocalizedResource.java
资源下载完成后，PublicLocalizer 驱动 LocalizedResource 向前

public class LocalizedResource implements EventHandler<ResourceEvent> {
  //跳变规则
  addTransition(ResourceState.DOWNLOADING, ResourceState.LOCALIZED,
                ResourceEventType.LOCALIZED, new FetchSuccessTransition())
  private static class FetchSuccessTransition extends ResourceTransition {
    @Override
    public void transition(LocalizedResource rsrc, ResourceEvent event) {
      //ResourceLocalizedEvent 对象
      ResourceLocalizedEvent locEvent = (ResourceLocalizedEvent) event;
      //搭载在 ResourceLocalizedEvent 上的资源路径名是个带 SchemeAndAuthority 的 URI
      rsrc.localPath =
          Path.getPathWithoutSchemeAndAuthority(locEvent.getLocation());
      rsrc.size = locEvent.getSize();
      for (ContainerId container : rsrc.ref) {
        //给所涉及的每个容器发送一个 ContainerEventType.RESOURCE _ LOCALIZED 事件
        //用此事件驱动 ContainerImpl 的状态机
        rsrc.dispatcher.getEventHandler().handle(
            new ContainerResourceLocalizedEvent(
              container, rsrc.rsrc, rsrc.localPath));
      }
    }
  }
}

server\nodemanager\containermanager\container\ContainerImpl.java

LocalizedResource 驱动 ContainerImpl 进一步驱动投运任务

public class ContainerImpl implements Container {
  //跳变规则
  addTransition(ContainerState.LOCALIZING,EnumSet.of(ContainerState.LOCALIZING, ContainerState.LOCALIZED),
                ContainerEventType.RESOURCE_LOCALIZED, new LocalizedTransition())
  static class LocalizedTransition implements
      MultipleArcTransition<ContainerImpl,ContainerEvent,ContainerState> {
    @SuppressWarnings("unchecked")
    @Override
    public ContainerState transition(ContainerImpl container,
        ContainerEvent event) {
      ContainerResourceLocalizedEvent rsrcEvent = (ContainerResourceLocalizedEvent) event;
      LocalResourceRequest resourceRequest = rsrcEvent.getResource();
      Path location = rsrcEvent.getLocation();
      //将事件中所载已完成本地化的资源(可有多项)从 pendingResources 集合中摘除
      List<String> syms = container.pendingResources.remove(resourceRequest);
      }
   
      container.sendLaunchEvent(); //容器中所有资源均已本地化,可以投运了
      container.metrics.endInitingContainer();

      return ContainerState.LOCALIZED;
    }
  }
}

ResourceLocalizationService.java
非公共资源处理，跨节点传输下载资源文件

 class LocalizerRunner extends Thread {

    @Override
    @SuppressWarnings("unchecked") // dispatcher not typed
    public void run() {
      Path nmPrivateCTokensPath = null;
      Throwable exception = null;
      try {
      // exec ： DefaultContainerExecutor
        if (dirsHandler.areDisksHealthy()) {
          exec.startLocalizer(new LocalizerStartContext.Builder()
              .setNmPrivateContainerTokens(nmPrivateCTokensPath)
              .setNmAddr(localizationServerAddress)
              .setUser(context.getUser())
              .setAppId(context.getContainerId()
                  .getApplicationAttemptId().getApplicationId().toString())
              .setLocId(localizerId)
              .setDirsHandler(dirsHandler)
              .build());
        }
    }
  }

server\nodemanager\DefaultContainerExecutor.java

public class DefaultContainerExecutor extends ContainerExecutor {
  @Override
  public void startLocalizer(LocalizerStartContext ctx)
      throws IOException, InterruptedException {
    Path nmPrivateContainerTokensPath = ctx.getNmPrivateContainerTokens();
    InetSocketAddress nmAddr = ctx.getNmAddr();
    String user = ctx.getUser();
    String appId = ctx.getAppId();
    String locId = ctx.getLocId();
    LocalDirsHandlerService dirsHandler = ctx.getDirsHandler();
    //目标目录路径的集合
    List<String> localDirs = dirsHandler.getLocalDirs();
    //用于 LOG 的目录路径集合
    List<String> logDirs = dirsHandler.getLogDirs();
    //在本地创建用户的目标文件目录
    createUserLocalDirs(localDirs, user);
    //创建该用户的缓存目录
    createUserCacheDirs(localDirs, user);
    //创建针对具体 AppId 的文件目录
    createAppDirs(localDirs, user, appId);
    //创建该 App 的日志文件目录
    createAppLogDirs(appId, logDirs, user);
    // 创建一个临时的工作目录
    Path appStorageDir = getWorkingDir(localDirs, user, appId);
    //进行本地化
    FileContext localizerFc = FileContext.getFileContext(
        lfs.getDefaultFileSystem(), getConf());
    localizerFc.setUMask(lfs.getUMask());
    //转入临时工作目录
    localizerFc.setWorkingDirectory(appStorageDir);
    //创建一个 ContainerLocalizer 类对象
    ContainerLocalizer localizer =
        createContainerLocalizer(user, appId, locId, localDirs, localizerFc);
    //调用 ContainerLocalizer.runLocalization ()
    localizer.runLocalization(nmAddr);
  }
}

server\nodemanager\containermanager\localizer\ContainerLocalizer.java

如何把非公共资源作为文件从另一个节点拷贝到本地公共资源的源头一定在RM 节点上,所以前面我们看到的本地化都是从 RM 节点拷贝文件。这比较简单,因为 NM节点与 RM 节点之间本来就有通信。但是非公共资源的源头却有可能在集群内的任何一个节点上,这就要复杂一些了,因为不同 NM 节点之间平时是没有通信的。正因为这样,我们需要与 资 源 所 在 的 节 点 临 时 建 立 通 信 管 道， 这 里 的 nodeManager 是 一 个 实 现 了LocalizationProtocol 界面的对象,实际上是 LocalizationProtocolPBClient Impl 。显然,这实际上是个采用 ProtoBuf 的 RPC 通道,但是它代表着对方，每一轮循环都以发送心跳报告开始,不过这不是向 RM 节点发送,而是向资源所在的 NM节点发送,所得的回应可以让我们知道对方是否还活着。如果还活着的话,就通过 download ()为需要从该结点下载的每项资源创建一个 Callable 对象 FSDownload 。

public class ContainerLocalizer {
 //主要的调用入口
 public void runLocalization(final InetSocketAddress nmAddr)
      throws IOException, InterruptedException {
     ......
    //获取通向对方 NodeManager 的 Proxy
    final LocalizationProtocol nodeManager =
        remoteUser.doAs(new PrivilegedAction<LocalizationProtocol>() {
          @Override
          public LocalizationProtocol run() {
            return getProxy(nmAddr); //返回 LocalizationProtocolPBClientImpl
          }
        });
    ExecutorService exec = null;
    try {
      //创建 ExecutorService 线程池
      exec = createDownloadThreadPool();
      //在线程池基础上创建 CompletionService ,用来接受和运行 callable
      CompletionService<Path> ecs = createCompletionService(exec);
      localizeFiles(nodeManager, ecs, ugi);
      return;
    } catch (Throwable e) {
      throw new IOException(e);
    }
  }
  
  //本地化文件
  protected void localizeFiles(LocalizationProtocol nodemanager,
      CompletionService<Path> cs, UserGroupInformation ugi)
      throws IOException, YarnException {
    while (true) {
      try {
        //逐项列出对彼岸节点的资源要求
        LocalizerStatus status = createStatus();
        //通过 PB 和 RPC 机制向对方发送心跳
        LocalizerHeartbeatResponse response = nodemanager.heartbeat(status);
        //根据对方回应分情形处理
        switch (response.getLocalizerAction()) {
        case LIVE:
          List<ResourceLocalizationSpec> newRsrcs = response.getResourceSpecs();
          //对于彼岸发回的每项资源描述
          for (ResourceLocalizationSpec newRsrc : newRsrcs) {
            if (!pendingResources.containsKey(newRsrc.getResource())) {
            //放在 pendingResources 集合中，又FSDownLoad下载，彼岸由LocalizerRunner接收处理
              pendingResources.put(newRsrc.getResource(), cs.submit(download(
                new Path(newRsrc.getDestinationDirectory().getFile()),
                newRsrc.getResource(), ugi)));
            }
          }
          break;
        cs.poll(1000, TimeUnit.MILLISECONDS);
      } 
    }
  }
}

server\nodemanager\containermanager\localizer\ResourceLocalizationService.java
非公共资源在NM与NM之间是通过心跳来保持通讯，LocalizerRunner来接收。这时候的心跳报告像个询问单,里面逐项列出了对本结点的资源要求。这里就在逐项考察这些资源是否存在,如果存在就为其创建一个资源描述,即 ResourceLocalizationSpec 。最后把这些资源描述搭载在心跳响应上发回给对方,让对方前来下载。

class LocalizerRunner extends Thread {
    //心跳接收
	LocalizerHeartbeatResponse processHeartbeat(List<LocalResourceStatus> remoteResourceStatuses) {
      LocalizerHeartbeatResponse response =
        recordFactory.newRecordInstance(LocalizerHeartbeatResponse.class);
      boolean fetchFailed = false;
      // 对于对方所要求的每项资源
      for (LocalResourceStatus stat : remoteResourceStatuses) {
        LocalResource rsrc = stat.getResource();
        LocalResourceRequest req = null;
        try {
          req = new LocalResourceRequest(rsrc);
        } catch (URISyntaxException e) {
        }
        LocalizerResourceRequestEvent assoc = scheduled.get(req);
        }
        LocalResourcesTracker tracker =
            getLocalResourcesTracker(req.getVisibility(), user, applicationId);
        }
        switch (stat.getStatus()) {
          case FETCH_SUCCESS:
            // notify resource
            try {
              tracker.handle(new ResourceLocalizedEvent(req,
                  stat.getLocalPath().toPath(), stat.getLocalSize()));
            } catch (URISyntaxException e) { }

            // unlocking the resource and removing it from scheduled resource
            // list
            assoc.getResource().unlock();
            scheduled.remove(req);
            break;
        }
      }
      // Give the localizer resources for remote-fetchin
      List<ResourceLocalizationSpec> rsrcs = new ArrayList<ResourceLocalizationSpec>();
      //查找本地资源
      LocalResource next = findNextResource();
      //资源存在,创建对于本项资源的描述
      if (next != null) {
        try {
          LocalResourcesTracker tracker = getLocalResourcesTracker(
              next.getVisibility(), user, applicationId);
          if (tracker != null) {
            Path localPath = getPathForLocalization(next, tracker);
            if (localPath != null) {
             //将资源描述加入 rsrcs 集合
              rsrcs.add(NodeManagerBuilderUtils.newResourceLocalizationSpec(
                  next, localPath));
            }
          }
        }
      }

      response.setLocalizerAction(LocalizerAction.LIVE);
      //把这些资源描述搭载在心跳响应上
      response.setResourceSpecs(rsrcs);
      return response;
    }
}

server\nodemanager\containermanager\container\ContainerImpl.java
非公共资源下载完成后，会执行sendLaunchEvent(), 向ContainersLauncher发送事件

public class ContainerImpl implements Container {
  @SuppressWarnings("unchecked") // dispatcher not typed
  private void sendLaunchEvent() {
    ContainersLauncherEventType launcherEvent =
        ContainersLauncherEventType.LAUNCH_CONTAINER;
    if (recoveredStatus == RecoveredContainerStatus.LAUNCHED) {
      // try to recover a container that was previously launched
      launcherEvent = ContainersLauncherEventType.RECOVER_CONTAINER;
    }
    containerLaunchStartTime = clock.getTime();
    dispatcher.getEventHandler().handle(
        new ContainersLauncherEvent(this, launcherEvent));
  }
}

server\nodemanager\containermanager\launcher\ContainersLauncher.java

public class ContainersLauncher extends AbstractService implements EventHandler<ContainersLauncherEvent> {
  @Override
  public void handle(ContainersLauncherEvent event) {
    // TODO: ContainersLauncher launches containers one by one!!
    Container container = event.getContainer();
    ContainerId containerId = container.getContainerId();
    switch (event.getType()) {
      case LAUNCH_CONTAINER:
        Application app =
          context.getApplications().get(
              containerId.getApplicationAttemptId().getApplicationId());
         //将 callable 对象 ContainerLaunch 提交给 ExecutorService
        ContainerLaunch launch =
            new ContainerLaunch(context, getConfig(), dispatcher, exec, app,
              event.getContainer(), dirsHandler, containerManager);
        containerLauncher.submit(launch);
        //并将其记录在 running 集合中
        running.put(containerId, launch);
        break;
   }
}

4. 容器投运

server\nodemanager\containermanager\launcher\ContainerLaunch.java

public class ContainerLaunch implements Callable<Integer> {
 public Integer call() { //调用入口
    //从容器中获取 CLC
    final ContainerLaunchContext launchContext = container.getLaunchContext();
    Map<Path,List<String>> localResources = null;
    ContainerId containerID = container.getContainerId();
    String containerIdStr = containerID.toString();
    //从 CLC 中获取命令行
    final List<String> command = launchContext.getCommands();
    int ret = -1;

    try {
      //资源本地化
      localResources = container.getLocalizedResources();
      final String user = container.getUser();
      // 创建新命令行,因为需要根据环境修正命令行中的一些元素
      List<String> newCmds = new ArrayList<String>(command.size());
      String appIdStr = app.getAppId().toString();
      String relativeContainerLogDir = ContainerLaunch
          .getRelativeContainerLogDir(appIdStr, containerIdStr);
      Path containerLogDir =
          dirsHandler.getLogPathForWrite(relativeContainerLogDir, false);
       //把老命令行中的元素逐一根据环境需要加以修正后转移到新命令行
      for (String str : command) {
        //加入新命令行
        newCmds.add(expandEnvironment(str, containerLogDir));
      }
      //把新命令行写回 CLC
      launchContext.setCommands(newCmds);

      Map<String, String> environment = launchContext.getEnvironment();
      // 逐一修正环境变量
      for (Entry<String, String> entry : environment.entrySet()) {
        String value = entry.getValue();
        value = expandEnvironment(value, containerLogDir);
        entry.setValue(value);
      }
      //决定采用何种文件系统
      FileContext lfs = FileContext.getLocalFSFileContext();
      //启动脚本路径,如“ xyz / launch _ container 
      Path nmPrivateContainerScriptPath =
          dirsHandler.getLocalPathForWrite(
              getContainerPrivateDir(appIdStr, containerIdStr) + Path.SEPARATOR
                  + CONTAINER_SCRIPT);
      // Token 文件路径
      Path nmPrivateTokensPath =
          dirsHandler.getLocalPathForWrite(
              getContainerPrivateDir(appIdStr, containerIdStr)
                  + Path.SEPARATOR
                  + String.format(ContainerLocalizer.TOKEN_FILE_NAME_FMT,
                      containerIdStr));
      //JAR 文件目录路径
      Path nmPrivateClasspathJarDir = 
          dirsHandler.getLocalPathForWrite(
              getContainerPrivateDir(appIdStr, containerIdStr));

      //工作目录路径
      Path containerWorkDir =
          dirsHandler.getLocalPathForWrite(ContainerLocalizer.USERCACHE
              + Path.SEPARATOR + user + Path.SEPARATOR
              + ContainerLocalizer.APPCACHE + Path.SEPARATOR + appIdStr
              + Path.SEPARATOR + containerIdStr,
              LocalDirAllocator.SIZE_UNKNOWN, false);
       // PID 文件子路径
      String pidFileSubpath = getPidFileSubpath(appIdStr, containerIdStr);

      //ID 文件全路径
      pidFilePath = dirsHandler.getLocalPathForWrite(pidFileSubpath);
      //本地目录
      List<String> localDirs = dirsHandler.getLocalDirs();
      //本地目录
      List<String> logDirs = dirsHandler.getLogDirs();
      List<String> containerLogDirs = new ArrayList<String>();
      for( String logDir : logDirs) {
        containerLogDirs.add(logDir + Path.SEPARATOR + relativeContainerLogDir);
      }
      //检查硬盘是否正常
      if (!dirsHandler.areDisksHealthy()) {
        ret = ContainerExitStatus.DISKS_FAILED;
        throw new IOException("Most of the disks failed. "
            + dirsHandler.getDisksHealthReport(false));
      }

      // 设置环境变量 HADOOP _ TOKEN _ FILE _ LOCATION， 变量值为工作目录下的“ container _ tokens 
      environment.put(
          ApplicationConstants.CONTAINER_TOKEN_FILE_ENV_NAME,
          new Path(containerWorkDir,
              FINAL_CONTAINER_TOKENS_FILE).toUri().getPath());

      // /// Write out the container-script in the nmPrivate space.
      try (DataOutputStream containerScriptOutStream =
               lfs.create(nmPrivateContainerScriptPath,
                   EnumSet.of(CREATE, OVERWRITE))) {
        // 设置一系列的环境变量
        sanitizeEnv(environment, containerWorkDir, appDirs, containerLogDirs,
          localResources, nmPrivateClasspathJarDir);

        // 把环境变量的设置,以及 launchContext 中带来的命令行写入脚本
        exec.writeLaunchEnv(containerScriptOutStream, environment,
          localResources, launchContext.getCommands(),
            new Path(containerLogDirs.get(0)));
      }
    
      //使 ContainerImpl 对象的状态机执行 LaunchTransition.transition (), 并进入 RUNNING 状态
      dispatcher.getEventHandler().handle(new ContainerEvent(
            containerID,
            ContainerEventType.CONTAINER_LAUNCHED));
      context.getNMStateStore().storeContainerLaunched(containerID);
      }
      else { //正常情况,发起容器运行
        exec.activateContainer(containerID, pidFilePath);
        //发起运行,这是个“blockingcall ”,当前线程将被阻塞
        //直至运行结束才会从这个函数调用返回
        ret = exec.launchContainer(new ContainerStartContext.Builder()
            .setContainer(container)
            .setLocalizedResources(localResources)
            .setNmPrivateContainerScriptPath(nmPrivateContainerScriptPath)
            .setNmPrivateTokensPath(nmPrivateTokensPath)
            .setUser(user)
            .setAppId(appIdStr)
            .setContainerWorkDir(containerWorkDir)
            .setLocalDirs(localDirs)
            .setLogDirs(logDirs)
            .build());
      }
    }
  }  
}

server\nodemanager\ContainerExecutor.java

public abstract class ContainerExecutor implements Configurable {
  @VisibleForTesting
  public void writeLaunchEnv(OutputStream out,
      Map<String, String> environment, Map<Path, List<String>> resources,
      List<String> command, Path logDir, String outFilename)
      throws IOException {
    updateEnvForWhitelistVars(environment);
    //构建Shell脚本
    ContainerLaunch.ShellScriptBuilder sb = ContainerLaunch.ShellScriptBuilder.create();
    //写入Command
    sb.command(command);
    PrintStream pout = null;
    try {
      pout = new PrintStream(out, false, "UTF-8");
      sb.write(pout);
    } finally {
      if (out != null) {
        out.close();
      }
    }
  }
}

server\nodemanager\DefaultContainerExecutor.java
以脚本的形式启动JVM

public class DefaultContainerExecutor extends ContainerExecutor {
  @Override
  public int launchContainer(ContainerStartContext ctx) throws IOException {
    Container container = ctx.getContainer();
    Shell.CommandExecutor shExec = null;
    try {
      setScriptExecutable(launchDst, user);
      setScriptExecutable(sb.getWrapperScriptPath(), user);

      shExec = buildCommandExecutor(sb.getWrapperScriptPath().toString(),
          containerIdStr, user, pidFile, container.getResource(),
          new File(containerWorkDir.toUri().getPath()),
          container.getLaunchContext().getEnvironment());
        if (isContainerActive(containerId)) {
        shExec.execute(); //让操作系统执行 shell 命令行
      }
    } 
    return 0;
  }
}

真正要启动执行的命令行则在容器脚本文件 launch _ container 中,那个脚本的核心就是由 ContainerLaunch Context 带来的命令行“ ~ / bin /java … MRAppMaster …”,就是要在宿主操作系统上运行 bin /java,即 Java 虚拟机,让其执行 MRAppMaster.class ,行 MRAppMaster.class 的 JVM 一经启动,当地就有了一个独立的 JVM 进程,这个JVM 从 MRAppMaster.main ()开始执行,这就是为所提交 App 在 Hadoop 集群上成立的“项目组”的组长。再往后就是 MRAppMaster 的事了, MRAppMaster 只是 MapReduce 计算的项目领导者, ApplicationMaster则是类似于 Unix / Linux 风格的流水线式计算的项目领导者。另外,这 里 我 们 要 在 目 标 NM 节 点 上 投 运 的 容 器 只 是 用 于 为 App 创 建 运 行 着MRAppMaster 或其他 AppMaster 的 JVM 进程。但是,同样的过程也可用于创建其他进程,由 ContainerLaunchContext 带来 NM 节点上的最核心的信息就是个 shell 命令行。