Hadoop源码解析之YARN客户端作业提交流程

1. 简介

hadoop在1.x中是向JobTracker提交，而在2.x中换成了ResourceManager，客户端的代理对象也有所变动，换成了YarnRunner，但大致流程和1类似，主要的流程集中在JobSubmitter.submitJobInternal中，包括检测输出目录合法性，设置作业提交信息（主机和用户），获得JobID，向HDFS中拷贝作业所需文件（Job.jar Job.xml split文件等），最后执行作业提交。

2.源码解析

waitForCompletion函数用于提交作业，循环监控作业状态

public boolean waitForCompletion(boolean verbose) throws IOException, InterruptedException,   ClassNotFoundException {  
  if (state == JobState.DEFINE) {  
    submit();//提交  
  }  
  if (verbose) {  
    monitorAndPrintJob();  
  } else {  
    // get the completion poll interval from the client.  
    int completionPollIntervalMillis =   Job.getCompletionPollInterval(cluster.getConf());  
    while (!isComplete()) {  
      try {  
        Thread.sleep(completionPollIntervalMillis);  
      } catch (InterruptedException ie) {  
      }  
    }  
  }  
  return isSuccessful();  
}  

主要分析submit函数，来看作业是如何提交的，主要分为两个阶段

1、连接master 

2、作业提交

public void submit() throws IOException, InterruptedException, ClassNotFoundException {  
  ensureState(JobState.DEFINE);  
  setUseNewAPI();  
  //连接RM  
  connect();  
  final JobSubmitter submitter =  getJobSubmitter(cluster.getFileSystem(), cluster.getClient());  
  status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {  
    public JobStatus run() throws IOException, InterruptedException,   ClassNotFoundException {  
        //提交作业  
      return submitter.submitJobInternal(Job.this, cluster);  
    }  
  });  
  state = JobState.RUNNING;  
  LOG.info("The url to track the job: " + getTrackingURL());  
}  

连接master时会建立Cluster实例，下面是Cluster构造函数，其中重点初始化部分

public Cluster(InetSocketAddress jobTrackAddr, Configuration conf)   throws IOException {  
  this.conf = conf;  
  this.ugi = UserGroupInformation.getCurrentUser();  
  initialize(jobTrackAddr, conf);  
}  

创建客户端代理阶段用到了java.util.ServiceLoader,目前2.5.2版本包含两个LocalClientProtocolProvider（本地作业） YarnClientProtocolProvider（Yarn作业），此处会根据mapreduce.framework.name的配置创建相应的客户端

private void initialize(InetSocketAddress jobTrackAddr, Configuration conf)  throws IOException {  
  synchronized (frameworkLoader) {  
    for (ClientProtocolProvider provider : frameworkLoader) {  
      LOG.debug("Trying ClientProtocolProvider : "  + provider.getClass().getName());  
      ClientProtocol clientProtocol = null;   
      try {  
        if (jobTrackAddr == null) {  
            //创建YARNRunner对象  
          clientProtocol = provider.create(conf);  
        } else {  
          clientProtocol = provider.create(jobTrackAddr, conf);  
        }  
        //初始化Cluster内部成员变量  
        if (clientProtocol != null) {  
          clientProtocolProvider = provider;  
          client = clientProtocol;  
          LOG.debug("Picked " + provider.getClass().getName()  + " as the ClientProtocolProvider");  
          break;  
        }  
        else {  
          LOG.debug("Cannot pick " + provider.getClass().getName()   + " as the ClientProtocolProvider - returned null protocol");  
        }  
      }   
      catch (Exception e) {  
        LOG.info("Failed to use " + provider.getClass().getName()  
            + " due to error: " + e.getMessage());  
      }  
    }  
  }  

  if (null == clientProtocolProvider || null == client) {  
    throw new IOException(  
        "Cannot initialize Cluster. Pleas