job提交的源码跟踪

job提交的源码跟踪

回顾总结提交作业的过程：

1.先执行方法job.waitForCompletion()，该方法是属于job类完成作业的提交并监控和打印job提交的过程。

在job.waitForCompletion()方法中，先判断job的状态是否为DEFINE，若为DEFINE则调用submit()方法，
其中submit()是该方法实现的核心。

在job.waitForCompletion()方法中，负责监控和打印job提交过程的方法是monitorAndPrintJob();

2.查看submit()方法的源码,该方法完成的功能是：

关键是创建submitter对象，并调用getJobSubmitter()方法，该方法中的重要参数是Cluster对象，不同的submiter对象
的创建是由该cluster对象决定的，Cluster对象是在connect()方法中创建的，连接客户端和服务器端connect()方法。
也检查了job的状态，设置为新的API等方法。

3.查看connect()方法中就是判断Cluster对象是否为空，Cluster对象是RPC Client对象，为空就创建一个Cluster对象
给其赋值。Cluster对象是如何创建的：

在connect()方法中调用了Cluster()，是Cluster对象的构造方法，在Cluster（）中重要的是initialize()方法。

4.查看initialize()的源码：

在该方法中我们可以看到通过遍历frameworkLoader的值，由此处也可以看到是本地模式还是yarn集群模式，
根据不同的frameworkLoader的值来调用不同的方法create(),我们从下图中可以看到有两个create()方法的

   
当我们选择LocalClientProtocolProvider时，此处为本地模式，该方法的返回值就是return new LocalJobRunner(conf);  

当选择YarnClientProtocolProvider时，此处为yarn集群模式，该方法的返回值就是 return new YARNRunner(conf);

只到此处我们可以看到Runner的创建。

在submit()方法中调用方法submitJobInternal()方法，判断作业的输出路径是否存在，获取jobId，创建路径

最后设置job的状态为RUNNING： state = JobState.RUNNING;

下面是job提交到yarn集群时源码跟踪：

job的提交是 job.waitForCompletion()方法起作用，产生一个提交器。在该处设置一个断点，可以跟踪该过程。

//以下是waitForCompletion的源码

    

public boolean waitForCompletion(boolean verbose
                                   ) throws IOException, InterruptedException,
                                            ClassNotFoundException {
// 查看job的状态(可以通过浏览器访问端口8088，查看All Application 页面已完成的job的状态为success但此时的状态应当为DEFINE)
    if (state == JobState.DEFINE) {
//将其job提交
      submit();
    }
//判断传入boolean参数是否为true，为true则打印出job的相关信息
    if (verbose) {
//打印监控和打印job提交的信息
      monitorAndPrintJob();
    } else {
      // get the completion poll interval from the client.
      int completionPollIntervalMillis = 
        Job.getCompletionPollInterval(cluster.getConf());
      while (!isComplete()) {
        try {
          Thread.sleep(completionPollIntervalMillis);
        } catch (InterruptedException ie) {
        }
      }
    }
    return isSuccessful();
  }

  跟踪submit()方法，源码如下：

  public void submit() throws IOException, InterruptedException, ClassNotFoundException {

    //检查job的状态
    ensureState(JobState.DEFINE);
    //设置新的API，因为版本的更新
    setUseNewAPI();
    //客户端和服务端的连接
    connect();
    //创建submitter,submitter的创建是根据getJobSubmitter()方法的参数区别创建的cluster
     由connect()方法产生的客户端
    final JobSubmitter submitter = getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
    status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {
      public JobStatus run() throws IOException, InterruptedException, 
      ClassNotFoundException {
//提交该作业，根据不同的cluster进行不同的作业(下面看一下connect()方法的实现过程)
        return submitter.submitJobInternal(Job.this, cluster);
      }
    });
//设置job的状态为RUNNING
    state = JobState.RUNNING;
    LOG.info("The url to track the job: " + getTrackingURL());</span>

   }

//connect()方法实现源码：

private synchronized void connect()
          throws IOException, InterruptedException, ClassNotFoundException {

               

  //判断cluster是否为空，为空则为cluster赋值
    if (cluster == null) {
      cluster = 
        ugi.doAs(new PrivilegedExceptionAction<Cluster>() {
                   public Cluster run()
                          throws IOException, InterruptedException, 
                                 ClassNotFoundException {
//此处通过Cluster的构造方法创建一个新的Cluster对象
                     return new Cluster(getConfiguration());
                   }
                 });
    }</span>

  }

 

在Cluster构造方法的源码中可以了解到：

 

public Cluster(InetSocketAddress jobTrackAddr, Configuration conf) 
      throws IOException {
    this.conf = conf;
    this.ugi = UserGroupInformation.getCurrentUser();
//在构造方法中重点是该方法中的实现
    initialize(jobTrackAddr, conf);
  }</span>

以下是  initialize()方法的实现的源码：

//clientProtocolProvider 动态代理对象，表示某种协议

//clientProtocolProvider 是为创建client对象

较为重要的是client、fs对象的创建

     

private void initialize(InetSocketAddress jobTrackAddr, Configuration conf)throws IOException {


    synchronized (frameworkLoader) {
//遍历provider,不同的provider来创建不同的client，frameworkLoader是 LocalJobRunner和YarnRunner
      for (ClientProtocolProvider provider : frameworkLoader) {
        LOG.debug("Trying ClientProtocolProvider : "
            + provider.getClass().getName());
        ClientProtocol clientProtocol = null; 
        try {
          if (jobTrackAddr == null) {
            clientProtocol = provider.create(conf);//由此进入create()方法，下面有create()方法的源码
          } else {
            clientProtocol = provider.create(jobTrackAddr, conf);//由此进入create()方法，下面有create()方法的源码
          }


          if (clientProtocol != null) {
            clientProtocolProvider = provider;
//在此处将clientProtocol赋值给client
            client = clientProtocol;
            LOG.debug("Picked " + provider.getClass().getName()
                + " as the ClientProtocolProvider");
            break;
          }
          else {
            LOG.debug("Cannot pick " + provider.getClass().getName()
                + " as the ClientProtocolProvider - returned null protocol");
          }
        } 
        catch (Exception e) {
          LOG.info("Failed to use " + provider.getClass().getName()
              + " due to error: " + e.getMessage());
        }
      }
    }


    if (null == clientProtocolProvider || null == client) {
      throw new IOException(
          "Cannot initialize Cluster. Please check your configuration for "
              + MRConfig.FRAMEWORK_NAME
              + " and the correspond server addresses.");
    }
  }</span>

//  clientProtocol = provider.create(conf);进入create()方法

    

public class LocalClientProtocolProvider extends ClientProtocolProvider {
public ClientProtocol create(Configuration conf) throws IOException {
//从配置文件中取出mapreduce.frame.name的值赋给framework，若未取到该值则赋值为local
    String framework =
        conf.get(MRConfig.FRAMEWORK_NAME, MRConfig.LOCAL_FRAMEWORK_NAME);
//判断当前的mapreduce.frame.name的值与framework比较，判断是否相等，若不相等则返回空
    if (!MRConfig.LOCAL_FRAMEWORK_NAME.equals(framework)) {
      return null;
    }
    conf.setInt(JobContext.NUM_MAPS, 1);
//若为二者相等创建LocalJobRunner，返回LocalJobRunner(conf);
    return new LocalJobRunner(conf);
  }</span>

//此处与LocalClientProtocolProvider相同

    

public class YarnClientProtocolProvider extends ClientProtocolProvider {


  @Override
  public ClientProtocol create(Configuration conf) throws IOException {
//仍是判断本地的mapreduce.frame.name的值与所取到的值是否相等
//可以查看YARN_FRAMEWORK_NAME   LOCAL_FRAMEWORK_NAME的值，对我们方便来读源码
//  public static final String YARN_FRAMEWORK_NAME  = "yarn";
//  public static final String LOCAL_FRAMEWORK_NAME = "local";
    if (MRConfig.YARN_FRAMEWORK_NAME.equals(conf.get(MRConfig.FRAMEWORK_NAME))) {
//创建一个YARNRunner
      return new YARNRunner(conf);
    }
    return null;
  }


  @Override
  public ClientProtocol create(InetSocketAddress addr, Configuration conf)
      throws IOException {
    return create(conf);
  }</span>

查看submitJobInternal()方法：

 

JobStatus submitJobInternal(Job job, Cluster cluster) 
  throws ClassNotFoundException, InterruptedException, IOException {


    //检查job作业完成后的输出目录
    checkSpecs(job);

//获取到job的配置参数
    Configuration conf = job.getConfiguration();
    addMRFrameworkToDistributedCache(conf);

//创建路径/tmp/hadoop-yarn/staging/hadoop/.staging
    Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
    // //获取客户端的ip地址，确定谁在提交
    InetAddress ip = InetAddress.getLocalHost();

    if (ip != null) {
      submitHostAddress = ip.getHostAddress();
      submitHostName = ip.getHostName();
      conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
      conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
    }
//获取jobId。通过submitClient（此时值为YarnRunner）客户端来与ResourceManager通信，获取jobId
    JobID jobId = submitClient.getNewJobID();
//设置job对象中的jobId值
    job.setJobID(jobId);
//用jobID创建出以唯一的路径
    Path submitJobDir = new Path(jobStagingArea, jobId.toString());
    JobStatus status = null;
    try {
      conf.set(MRJobConfig.USER_NAME,
          UserGroupInformation.getCurrentUser().getShortUserName());
      conf.set("hadoop.http.filter.initializers", 
          "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer");
      conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, submitJobDir.toString());
      LOG.debug("Configuring job " + jobId + " with " + submitJobDir 
          + " as the submit dir");
      // get delegation token for the dir
      TokenCache.obtainTokensForNamenodes(job.getCredentials(),
          new Path[] { submitJobDir }, conf);
      
      populateTokenCache(conf, job.getCredentials());


      // generate a secret to authenticate shuffle transfers 缓存信息
      if (TokenCache.getShuffleSecretKey(job.getCredentials()) == null) {
        KeyGenerator keyGen;
        try {
          keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM);
          keyGen.init(SHUFFLE_KEY_LENGTH);
        } catch (NoSuchAlgorithmException e) {
          throw new IOException("Error generating shuffle secret key", e);
        }
        SecretKey shuffleKey = keyGen.generateKey();
        TokenCache.setShuffleSecretKey(shuffleKey.getEncoded(),
            job.getCredentials());
      }

//向我们用jobId关联的路径内写入资源，包含写入副本的信息，此处相当于客户端定义的副本数优先级优于服务器端，拷贝jar包
      copyAndConfigureFiles(job, submitJobDir);
      Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir);
      
      // Create the splits for the job
      LOG.debug("Creating splits at " + jtFs.makeQualified(submitJobDir));
//划分切数据片，来划分map,并将划分的策略写入一个文件，也会提交
      int maps = writeSplits(job, submitJobDir);
//将map写入conf中
      conf.setInt(MRJobConfig.NUM_MAPS, maps);
      LOG.info("number of splits:" + maps);


      // write "queue admins of the queue to which job is being submitted"
      // to job file.
      String queue = conf.get(MRJobConfig.QUEUE_NAME,
          JobConf.DEFAULT_QUEUE_NAME);
      AccessControlList acl = submitClient.getQueueAdmins(queue);
      conf.set(toFullPropertyName(queue,
          QueueACL.ADMINISTER_JOBS.getAclName()), acl.getAclString());


      // removing jobtoken referrals before copying the jobconf to HDFS
      // as the tasks don't need this setting, actually they may break
      // because of it if present as the referral will point to a
      // different job.
      TokenCache.cleanUpTokenReferral(conf);


      if (conf.getBoolean(
          MRJobConfig.JOB_TOKEN_TRACKING_IDS_ENABLED,
          MRJobConfig.DEFAULT_JOB_TOKEN_TRACKING_IDS_ENABLED)) {
        // Add HDFS tracking ids
        ArrayList<String> trackingIds = new ArrayList<String>();
        for (Token<? extends TokenIdentifier> t :
            job.getCredentials().getAllTokens()) {
          trackingIds.add(t.decodeIdentifier().getTrackingId());
        }
        conf.setStrings(MRJobConfig.JOB_TOKEN_TRACKING_IDS,
            trackingIds.toArray(new String[trackingIds.size()]));
      }


      // Write job file to submit dir
      writeConf(conf, submitJobFile);
      
      //
      // Now, actually submit the job (using the submit name)
      //
      printTokens(jobId, job.getCredentials());
      status = submitClient.submitJob(
          jobId, submitJobDir.toString(), job.getCredentials());
      if (status != null) {
        return status;
      } else {
        throw new IOException("Could not launch job");
      }
    } finally {
      if (status == null) {
        LOG.info("Cleaning up the staging area " + submitJobDir);
        if (jtFs != null && submitJobDir != null)
          jtFs.delete(submitJobDir, true);


      }
    }
  }