Hadoop-MapReduce源码分析

留不住的人

已于 2023-04-04 17:01:52 修改

阅读量269

点赞数 1

分类专栏：源码研究院文章标签： mapreduce hadoop hdfs

于 2023-04-04 16:46:28 首次发布

本文链接：https://blog.csdn.net/manba_yqq/article/details/129956391

版权

源码研究院专栏收录该内容

19 篇文章 3 订阅

订阅专栏

MapReduce源码分析

1.1 准备工作

在WCMapper类中的map方法的首行添加如下代码：
```
Thread.sleep(99999999);
```
重新打jar包

上传到hadoop集群中，重新运行

yarn jar wc.jar com.itbaizhan.WCDriver /wordcount/input /wordcount/output3

在hadoop集群中的任何节点上执行如下命令：

[root@node2 ~]# hdfs dfs -ls -R /tmp/hadoop-yarn/
drwx------   - root supergroup          0 2021-10-29 03:49 /tmp/hadoop-yarn/staging/root/.staging/job_1635443832663_0002

下载文件夹/tmp/hadoop-yarn/staging/root/.staging/job_1635443832663_0002

[root@node2 ~]# hdfs dfs -get /tmp/hadoop-yarn/staging/root/.staging/job_1635443832663_0002

从该节点的/root目录下下载到windows系统的桌面上，内容列表如下

在这里插入图片描述

job.jar:作业的jar包

job.xml：当前作业的参数的配置文件

job.split和job.splitmetainfo：当前作业的逻辑切片的相关信息文件

将job.xml拷贝到wordcount项目的根目录，然后进行格式化（目的：方便查看参数），使用Ctrl+Alt+L进行格式化。
配置信息的来源：

默认配置信息：

一部分来源于MRJobConfig接口

一部分来至于*-default.xml文件

自定义的信息：

一部分来至于xxx-site.xml文件

一部分来至于我们通过程序设置的参数

1.2 客户端提交作业

点击WCDriver类中的waitForCompletion方法

boolean result = job.waitForCompletion(true);
//或者
job.waitForCompletion(true);

waitForCompletion方法的内容如下：


  /**提交作业并等待作业的完成
   * Submit the job to the cluster and wait for it to finish.
   * @param verbose print the progress to the user：
   * @return true if the job succeeded
   * @throws IOException thrown if the communication with the 
   *         <code>JobTracker</code> is lost
   */
  public boolean waitForCompletion(boolean verbose
                                   ) throws IOException, InterruptedException,
                                            ClassNotFoundException {
    if (state == JobState.DEFINE) {
      submit();//提交作业
    }
    if (verbose) {//true会调用 monitorAndPrintJob();
      //打印作业的进度的相关日志
      monitorAndPrintJob();
    } else {
      //每隔5000ms指定的时间判断作业是否完成
      // get the completion poll interval from the client.
      int completionPollIntervalMillis = 
        Job.getCompletionPollInterval(cluster.getConf());
      while (!isComplete()) {
        try {
          Thread.sleep(completionPollIntervalMillis);
        } catch (InterruptedException ie) {
        }
      }
    }
    return isSuccessful();
  }

monitorAndPrintJob方法源码分析


  /**
   * Monitor a job and print status in real-time as progress is made and tasks 
   * fail.
   * @return true if the job succeeded
   * @throws IOException if communication to the JobTracker fails
   */
  public boolean monitorAndPrintJob() 
      throws IOException, InterruptedException {
    String lastReport = null;
    Job.TaskStatusFilter filter;
      //获取作业的配置文件对象
    Configuration clientConf = getConfiguration();
    filter = Job.getTaskOutputFilter(clientConf);
      //获取作业的id
    JobID jobId = getJobID();
      //输出作业Id
    LOG.info("Running job: " + jobId);
    int eventCounter = 0;
    boolean profiling = getProfileEnabled();
    IntegerRanges mapRanges = getProfileTaskRange(true);
    IntegerRanges reduceRanges = getProfileTaskRange(false);
      //每隔1000ms判断一次作业是否完成
    int progMonitorPollIntervalMillis = 
      Job.getProgressPollInterval(clientConf);
    /* make sure to report full progress after the job is done */
    boolean reportedAfterCompletion = false;
    boolean reportedUberMode = false;
    while (!isComplete() || !reportedAfterCompletion) {
      if (isComplete()) {
        reportedAfterCompletion = true;
      } else {
          //休眠1000ms
        Thread.sleep(progMonitorPollIntervalMillis);
      }
      if (status.getState() == JobStatus.State.PREP) {
        continue;
      }      
      if (!reportedUberMode) {
        reportedUberMode = true;
        LOG.info("Job " + jobId + " running in uber mode : " + isUber());
      }      
      String report = 
        (" map " + StringUtils.formatPercent(mapProgress(), 0)+
            " reduce " + 
            StringUtils.formatPercent(reduceProgress(), 0));
        //本次报告的执行进度和上次报告的进度不同才会打印
      if (!report.equals(lastReport)) {
        LOG.info(report);
        lastReport = report;
      }

      TaskCompletionEvent[] events = 
        getTaskCompletionEvents(eventCounter, 10); 
      eventCounter += events.length;
      printTaskEvents(events, filter, profiling, mapRanges, reduceRanges);
    }
    boolean success = isSuccessful();
      //输出作业的最终执行信息
    if (success) {
      LOG.info("Job " + jobId + " completed successfully");
    } else {
      LOG.info("Job " + jobId + " failed with state " + status.getState() + 
          " due to: " + status.getFailureInfo());
    }
      //相关计数器信息的处理
    Counters counters = getCounters();
    if (counters != null) {
      LOG.info(counters.toString());
    }
    return success;
  }

submit()方法分析（在waitForCompletion方法中点击submit()进入）


  /**
   * Submit the job to the cluster and return immediately.
   * @throws IOException
   */
  public void submit() 
         throws IOException, InterruptedException, ClassNotFoundException {
    ensureState(JobState.DEFINE);
    setUseNewAPI();
    connect();
    final JobSubmitter submitter = 
        getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
    status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {
      public JobStatus run() throws IOException, InterruptedException, 
      ClassNotFoundException {
          //进行作业的提交
        return submitter.submitJobInternal(Job.this, cluster);
      }
    });
    state = JobState.RUNNING;
    LOG.info("The url to track the job: " + getTrackingURL());
   }

submitJobInternal()方法提交作业（从submit()方法进入的）


  /**
   * Internal method for submitting jobs to the system.The job submission process involves:
   * 1.Checking the input and output specifications of the job.
      检查作业的输入输出路径
   * 2.Computing the {@link InputSplit}s for the job.
      计算作业的切片信息
   * 3.Setup the requisite accounting information for the 
   *   {@link DistributedCache} of the job, if necessary.
     如果需要，给分布式缓存设置比较的计数器
   * 4.Copying the job's jar and configuration to the map-reduce system
   *   directory on the distributed file-system. 
     将jar包、配置文件、切片信息等提交到hdfs上map-reduce系统目录
   * 5.Submitting the job to the <code>JobTracker</code> and optionally
   *   monitoring it's status.
   	 将作业提交给JobTracker(hadoop1.x)/ResourceManager(hadoop2.x+),并可选的监控作业的状态。
   * @param job the configuration to submit
   * @param cluster the handle to the Cluster
   * @throws ClassNotFoundException
   * @throws InterruptedException
   * @throws IOException
   */
  JobStatus submitJobInternal(Job job, Cluster cluster) 
  throws ClassNotFoundException, InterruptedException, IOException {
	//验证作业的输出路径
    //validate the jobs output specs 
    checkSpecs(job);

    Configuration conf = job.getConfiguration();
    addMRFrameworkToDistributedCache(conf);
	//作业的相关文件(jar,xml,split等)提交到路径
    Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
    //configure the command line options correctly on the submitting dfs
    InetAddress ip = InetAddress.getLocalHost();
    if (ip != null) {
      submitHostAddress = ip.getHostAddress();
      submitHostName = ip.getHostName();
      conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
      conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
    }
    JobID jobId = submitClient.getNewJobID();
    job.setJobID(jobId);
    Path submitJobDir = new Path(jobStagingArea, jobId.toString());
    JobStatus status = null;
    try {
      conf.set(MRJobConfig.USER_NAME,
          UserGroupInformation.getCurrentUser().getShortUserName());
      conf.set("hadoop.http.filter.initializers", 
          "org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer");
      conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, submitJobDir.toString());
      LOG.debug("Configuring job " + jobId + " with " + submitJobDir 
          + " as the submit dir");
      // get delegation token for the dir
      TokenCache.obtainTokensForNamenodes(job.getCredentials(),
          new Path[] { submitJobDir }, conf);
      
      populateTokenCache(conf, job.getCredentials());

      // generate a secret to authenticate shuffle transfers
      if (TokenCache.getShuffleSecretKey(job.getCredentials()) == null) {
        KeyGenerator keyGen;
        try {
          keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM);
          keyGen.init(SHUFFLE_KEY_LENGTH);
        } catch (NoSuchAlgorithmException e) {
          throw new IOException("Error generating shuffle secret key", e);
        }
        SecretKey shuffleKey = keyGen.generateKey();
        TokenCache.setShuffleSecretKey(shuffleKey.getEncoded(),
            job.getCredentials());
      }
      if (CryptoUtils.isEncryptedSpillEnabled(conf)) {
        conf.setInt(MRJobConfig.MR_AM_MAX_ATTEMPTS, 1);
        LOG.warn("Max job attempts set to 1 since encrypted intermediate" +
                "data spill is enabled");
      }

      copyAndConfigureFiles(job, submitJobDir);

      Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir);
      
      // Create the splits for the job
      LOG.debug("Creating splits at " + jtFs.makeQualified(submitJobDir));
      int maps = writeSplits(job, submitJobDir);
      conf.setInt(MRJobConfig.NUM_MAPS, maps);
      LOG.info("number of splits:" + maps);

      int maxMaps = conf.getInt(MRJobConfig.JOB_MAX_MAP,
          MRJobConfig.DEFAULT_JOB_MAX_MAP);
      if (maxMaps >= 0 && maxMaps < maps) {
        throw new IllegalArgumentException("The number of map tasks " + maps +
            " exceeded limit " + maxMaps);
      }

      // write "queue admins of the queue to which job is being submitted"
      // to job file.
      String queue = conf.get(MRJobConfig.QUEUE_NAME,
          JobConf.DEFAULT_QUEUE_NAME);
      AccessControlList acl = submitClient.getQueueAdmins(queue);
      conf.set(toFullPropertyName(queue,
          QueueACL.ADMINISTER_JOBS.getAclName()), acl.getAclString());

      // removing jobtoken referrals before copying the jobconf to HDFS
      // as the tasks don't need this setting, actually they may break
      // because of it if present as the referral will point to a
      // different job.
      TokenCache.cleanUpTokenReferral(conf);

      if (conf.getBoolean(
          MRJobConfig.JOB_TOKEN_TRACKING_IDS_ENABLED,
          MRJobConfig.DEFAULT_JOB_TOKEN_TRACKING_IDS_ENABLED)) {
        // Add HDFS tracking ids
        ArrayList<String> trackingIds = new ArrayList<String>();
        for (Token<? extends TokenIdentifier> t :
            job.getCredentials().getAllTokens()) {
          trackingIds.add(t.decodeIdentifier().getTrackingId());
        }
        conf.setStrings(MRJobConfig.JOB_TOKEN_TRACKING_IDS,
            trackingIds.toArray(new String[trackingIds.size()]));
      }

      // Set reservation info if it exists
      ReservationId reservationId = job.getReservationId();
      if (reservationId != null) {
        conf.set(MRJobConfig.RESERVATION_ID, reservationId.toString());
      }

      // Write job file to submit dir
      writeConf(conf, submitJobFile);
      
      //
      // Now, actually submit the job (using the submit name)
      //
      printTokens(jobId, job.getCredentials());
      status = submitClient.submitJob(
          jobId, submitJobDir.toString(), job.getCredentials());
      if (status != null) {
        return status;
      } else {
        throw new IOException("Could not launch job");
      }
    } finally {
      if (status == null) {
        LOG.info("Cleaning up the staging area " + submitJobDir);
        if (jtFs != null && submitJobDir != null)
          jtFs.delete(submitJobDir, true);

      }
    }
  }

分析job资源提交的hdfs上的路径（选学）

入口：

//前置路径
Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
//configure the command line options correctly on the submitting dfs
InetAddress ip = InetAddress.getLocalHost();
if (ip != null) {
    submitHostAddress = ip.getHostAddress();
    submitHostName = ip.getHostName();
    conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
    conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
}
JobID jobId = submitClient.getNewJobID();
job.setJobID(jobId);
//提交作业相关文件的hdfs上具体路径
Path submitJobDir = new Path(jobStagingArea, jobId.toString());

前置路径：

public static Path getStagingDir(Cluster cluster, Configuration conf)
    throws IOException, InterruptedException {
    UserGroupInformation user = UserGroupInformation.getLoginUser();
    //cluster:集群对象, conf：配置文件对象, user：当前用户 root
    return getStagingDir(cluster, conf, user);
}
public static Path getStagingDir(Cluster cluster, Configuration conf,
      UserGroupInformation realUser) throws IOException, InterruptedException {
    //获取对应前置路径 
    Path stagingArea = cluster.getStagingAreaDir();
    FileSystem fs = stagingArea.getFileSystem(conf);
    UserGroupInformation currentUser = realUser.getCurrentUser();
    try {
      ....
    } catch (FileNotFoundException e) {
        //在hdfs创建对应的目录
      fs.mkdirs(stagingArea, new FsPermission(JOB_DIR_PERMISSION));
    }
    return stagingArea;
}

public static Path getStagingAreaDir(Configuration conf, String user) {
        return new Path(conf.get("yarn.app.mapreduce.am.staging-dir", "/tmp/hadoop-yarn/staging") + "/" + user + "/" + ".staging");
    }

获取到前置路径如下：/tmp/hadoop-yarn/staging/root/.staging

所以最终路径为：/tmp/hadoop-yarn/staging/root/.staging/jobId

1.3 切片数量计算（重点）

程序入口JobSubmitter类中的submitJobInternal方法的198行开始

JobStatus submitJobInternal(Job job, Cluster cluster) 
throws ClassNotFoundException, InterruptedException, IOException {
	// Create the splits for the job   198行
    LOG.debug("Creating splits at " + jtFs.makeQualified(submitJobDir));
    //写切片信息文件到/tmp/hadoop-yarn/staging/root/.staging/jobId下，并返回切片的数量
    int maps = writeSplits(job, submitJobDir);
    //使用切片数量作为MapTask任务的数量，所以说有几个切片就对应几个MapTask
    conf.setInt(MRJobConfig.NUM_MAPS, maps);
    LOG.info("number of splits:" + maps);
}

切片计算调用的哪个方法进行计算的？

private int writeSplits(org.apache.hadoop.mapreduce.JobContext job,
        Path jobSubmitDir) throws IOException,InterruptedException, 
		ClassNotFoundException {
    JobConf jConf = (JobConf)job.getConfiguration();
    int maps;
    if (jConf.getUseNewMapper()) {
        //hadoop2.x+ 调用该方法进行前切片的计算
        maps = writeNewSplits(job, jobSubmitDir);
    } else {
        //hadoop1.x使用的切片计算的方法
        maps = writeOldSplits(jConf, jobSubmitDir);
    }
    return maps;
}

分析writeNewSplits(job, jobSubmitDir)方法，研究切片计算的细节。

private <T extends InputSplit>
  int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
      InterruptedException, ClassNotFoundException {
    Configuration conf = job.getConfiguration();
    //确定InputFormat类使用是？ 通过这个类就可以知道如何截取的切片     
    InputFormat<?, ?> input =
      ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
    //切片的计算逻辑
    List<InputSplit> splits = input.getSplits(job);
    //array数组中保存是InputSplit对象，也就是切片对象
    T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);

    // sort the splits into order based on size, so that the biggest
    // go first   对数组中的切片对象进行排序
    Arrays.sort(array, new SplitComparator());
    //向指定的目录下写切片信息 job.split和job.splitmetainfo         
    JobSplitWriter.createSplitFiles(jobSubmitDir, conf, 
        jobSubmitDir.getFileSystem(conf), array);
    //array数组的长度就是切片的数量（MapTask的数量）      
    return array.length;
}

由于input对象时抽象类InputFormat声明的，点入getSplits方法时，是一个抽象方法：

public abstract class InputFormat<K, V> {

  public abstract 
    List<InputSplit> getSplits(JobContext context
                               ) throws IOException, InterruptedException;
}

选中方法名getSplits，然后Ctrl+Alt+B,弹出如下界面：

在这里插入图片描述

确定InputFormat类是哪个？

Ctrl+点击步骤3中job.getInputFormatClass()进行类的确定

  public Class<? extends InputFormat<?,?>> getInputFormatClass() 
     throws ClassNotFoundException;

选择getInputFormatClass(),Ctrl+Alt+B->选择JobContextImpl类中的代码：

public Class<? extends InputFormat<?,?>> getInputFormatClass() 
    throws ClassNotFoundException {
    //INPUT_FORMAT_CLASS_ATTR = "mapreduce.job.inputformat.class" 

    return (Class<? extends InputFormat<?,?>>) 
        conf.getClass(INPUT_FORMAT_CLASS_ATTR, TextInputFormat.class);
}

点击conf.getClass()

public Class<?> getClass(String name, Class<?> defaultValue) {
    String valueString = getTrimmed(name);
    if (valueString == null)
      return defaultValue;
    try {
      return getClassByName(valueString);
    } catch (ClassNotFoundException e) {
      throw new RuntimeException(e);
    }
}

如果如上方法代码后发现：如果之前指定过mapreduce.job.inputformat.class 对应的format类使用自己指定的，如果没有指定过使用默认的TextInputFormat。

进入TextInputFormat类，

点击、或 Ctrl+Shift+R->输入TextInputFormat，进入：

package org.apache.hadoop.mapreduce.lib.input;
...
import com.google.common.base.Charsets;

@InterfaceAudience.Public
@InterfaceStability.Stable
public class TextInputFormat extends FileInputFormat<LongWritable, Text> {

  @Override
  public RecordReader<LongWritable, Text> 
    createRecordReader(InputSplit split,
                       TaskAttemptContext context) {
    String delimiter = context.getConfiguration().get(
        "textinputformat.record.delimiter");
    byte[] recordDelimiterBytes = null;
    if (null != delimiter)
      recordDelimiterBytes = delimiter.getBytes(Charsets.UTF_8);
    return new LineRecordReader(recordDelimiterBytes);
  }

  @Override
  protected boolean isSplitable(JobContext context, Path file) {
    final CompressionCodec codec =
      new CompressionCodecFactory(context.getConfiguration()).getCodec(file);
    if (null == codec) {
      return true;
    }
    return codec instanceof SplittableCompressionCodec;
  }

}

进入后发现该类父类为FileInputFormat类，该类对getSplits方法进行了重写。

进入FileInputFormat类，阅读它的getSplits方法：

block块是物理上进行的切割，而切片是逻辑上的概念，切片需要确定以下几点：

A. 是哪个文件的切片

B.切片从哪里开始（相对原文件的偏移量）

C.切片在哪个主机上

D.切片的大小

public List<InputSplit> getSplits(JobContext job) throws IOException {
    StopWatch sw = new StopWatch().start();
    //被用于计算切片的大小  minsize=max(1,0)=1  maxsize=Long.MAX_VALUE
    long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));
    long maxSize = getMaxSplitSize(job);

    // generate splits
    List<InputSplit> splits = new ArrayList<InputSplit>();
    List<FileStatus> files = listStatus(job);

    boolean ignoreDirs = !getInputDirRecursive(job)
      && job.getConfiguration().getBoolean(INPUT_DIR_NONRECURSIVE_IGNORE_SUBDIRS, false);
    //输入路径有些时候是目录，下面有多个被处理的文件，所以需要遍历
    for (FileStatus file: files) {
      if (ignoreDirs && file.isDirectory()) {
        continue;
      }
      //获取对应文件的对象  
      Path path = file.getPath();
      //获取文件的总大小  
      long length = file.getLen();
      //表示如果指定的输入路径下的文件大小为0(为空文件),不进行分析。  
      if (length != 0) {
        BlockLocation[] blkLocations;
        if (file instanceof LocatedFileStatus) {
          blkLocations = ((LocatedFileStatus) file).getBlockLocations();
        } else {
          FileSystem fs = path.getFileSystem(job.getConfiguration());
          blkLocations = fs.getFileBlockLocations(file, 0, length);
        }
        if (isSplitable(job, path)) {
          //获取block块的大小，hadoop2.x+默认是128MB,hadoop1.x默认64MB  
          long blockSize = file.getBlockSize();
          //计算的切片的大小minsize=1  maxsize=Long.MAX_VALUE blockSize=128*1024*1024 
          long splitSize = computeSplitSize(blockSize, minSize, maxSize);
		  /*protected long computeSplitSize(long blockSize, long minSize,
                                  long maxSize) {
               //所以计算之后splitSize默认=blockSize=128MB,一个block块对应split切片     
               return Math.max(minSize, Math.min(maxSize, blockSize));
            }
		  */
          /* 如果调整切片的大小：
              1.设置为256MB: 只需要将minSize设置256*1024*1024B
              conf.set(FileInputFormat.SPLIT_MINSIZE,"268435456");//256MB
              2.设置为64MB: 只需要将maxSize设置为64*1024*1024
              conf.set(FileInputFormat.SPLIT_MAXSIZE,"67108864");//64MB
          */  
          //首先将文件的大小赋值给bytesRemaining（表示的是剩余可切的文件大小）  
          long bytesRemaining = length;
          //SPLIT_SLOP=1.1  
          while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, splitSize,
                        blkLocations[blkIndex].getHosts(),
                        blkLocations[blkIndex].getCachedHosts()));
            bytesRemaining -= splitSize;
          }

          if (bytesRemaining != 0) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
                       blkLocations[blkIndex].getHosts(),
                       blkLocations[blkIndex].getCachedHosts()));
          }
        } else { // not splitable
          //如果文件不可切，将整个文件作为一个切片对象添加到splits集合中  
          if (LOG.isDebugEnabled()) {
            // Log only if the file is big enough to be splitted
            if (length > Math.min(file.getBlockSize(), minSize)) {
              LOG.debug("File is not splittable so no parallelization "
                  + "is possible: " + file.getPath());
            }
          }
          splits.add(makeSplit(path, 0, length, blkLocations[0].getHosts(),
                      blkLocations[0].getCachedHosts()));
        }
      } else { 
        //Create empty hosts array for zero length files
        splits.add(makeSplit(path, 0, length, new String[0]));
      }
    }
    // Save the number of input files for metrics/loadgen
    job.getConfiguration().setLong(NUM_INPUT_FILES, files.size());
    sw.stop();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Total # of splits generated by getSplits: " + splits.size()
          + ", TimeTaken: " + sw.now(TimeUnit.MILLISECONDS));
    }
    return splits;
  }

如何修改使用的InputFormat类？

默认使用是TextInputFormat,如何修改？

private <T extends InputSplit>
  int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
      InterruptedException, ClassNotFoundException {
    Configuration conf = job.getConfiguration();
    InputFormat<?, ?> input =
      ReflectionUtils.newInstance(job.getInputFormatClass(), conf);
    List<InputSplit> splits = input.getSplits(job);
  }

点击：getInputFormatClass（）方法，进入JobContext接口中：

  public Class<? extends InputFormat<?,?>> getInputFormatClass() 
     throws ClassNotFoundException;

选中getInputFormatClass()方法，Ctrl+Alt+B，选中JobContextImpl

  public Class<? extends InputFormat<?,?>> getInputFormatClass() 
     throws ClassNotFoundException {
    return (Class<? extends InputFormat<?,?>>) 
      conf.getClass(INPUT_FORMAT_CLASS_ATTR, TextInputFormat.class);
  }

通过conf对象从配置文件中找常量INPUT_FORMAT_CLASS_ATTR对应的字符串mapreduce.job.inputformat.class为key的值，如果没有设置过，则使用默认的TextInputFormat。

修改的话有两种方式：

//方式一：常用的方式
job.setInputFormatClass(KeyValueTextInputFormat.class);
//方式二：不常用
conf.set("mapreduce.job.inputformat.class","org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat");

1.4 Map阶段源码分析

在这里插入图片描述

从切片读取数据，给map方法，map方法输出的键值对写到环形缓冲区，关注以下内容：

在写到环形缓冲区之前计算分区号
环形缓冲区排序后溢写到map端本地磁盘，可能会有合并的过程，3
可以设置环形缓冲区大小和阈值
排序所使用的算法：默认是快排，可以设置
可以自定义key和value
排序比较器可以自定义
可以设置Combiner类

1.4.1 源码从哪里看？

在这里插入图片描述

从上图可知，MapTask入口从YarnChild类开始，在IDEA中Ctrl+Shift+R->YarnChild->main()->174行开始阅读。

Ctrl+Alt+<- :返回到上一个位置

Ctrl+Alt±>:回到下一个位置

childUGI.doAs(new PrivilegedExceptionAction<Object>() {
    @Override
    public Object run() throws Exception {
        // use job-specified working directory
        setEncryptedSpillKeyIfRequired(taskFinal);
        FileSystem.get(job).setWorkingDirectory(job.getWorkingDirectory());
        taskFinal.run(job, umbilical); // run the task  Ctrl+单击该run()方法
        return null;
    }
});

1.4.2 runNewMapper()方法引入

进入Task.java类的run()方法：

public abstract void run(JobConf job, TaskUmbilicalProtocol umbilical)
    throws IOException, ClassNotFoundException, InterruptedException;

选中run()方法之后Ctrl+Alt+B->选中MapTask ->run()方法中。

public void run(final JobConf job, final TaskUmbilicalProtocol umbilical)
    throws IOException, ClassNotFoundException, InterruptedException {
    this.umbilical = umbilical;
    //如果为MapTask,则继续
    if (isMapTask()) {
      // If there are no reducers then there won't be any sort. Hence the map 
      // phase will govern the entire attempt's progress.
      //获取ReduceTask的数量  
      if (conf.getNumReduceTasks() == 0) {//后续没有reduce任务
        mapPhase = getProgress().addPhase("map", 1.0f);
      } else {//后续有reduce任务
        // If there are reducers then the entire attempt's progress will be 
        // split between the map phase (67%) and the sort phase (33%).
        mapPhase = getProgress().addPhase("map", 0.667f);
        sortPhase  = getProgress().addPhase("sort", 0.333f);
      }
    }
    TaskReporter reporter = startReporter(umbilical);
    //获取是否使用新版的MapperApi
    boolean useNewApi = job.getUseNewMapper();
    initialize(job, getJobID(), reporter, useNewApi);

    // check if it is a cleanupJobTask
    if (jobCleanup) {
      runJobCleanupTask(umbilical, reporter);
      return;
    }
    if (jobSetup) {
      runJobSetupTask(umbilical, reporter);
      return;
    }
    if (taskCleanup) {
      runTaskCleanupTask(umbilical, reporter);
      return;
    }

    if (useNewApi) {//hadoop2.x+使用
      runNewMapper(job, splitMetaInfo, umbilical, reporter);
    } else {//hadoop1.x使用
      runOldMapper(job, splitMetaInfo, umbilical, reporter);
    }
    done(umbilical, reporter);
  }

1.4.3 runNewMapper()方法分析

private <INKEY,INVALUE,OUTKEY,OUTVALUE>
  void runNewMapper(final JobConf job,
                    final TaskSplitIndex splitIndex,
                    final TaskUmbilicalProtocol umbilical,
                    TaskReporter reporter
                    ) throws IOException, ClassNotFoundException,
                             InterruptedException {
    // make a task context so we can get the classes
    org.apache.hadoop.mapreduce.TaskAttemptContext taskContext =
      new org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(job, 
                                                                  getTaskID(),
                                                                  reporter);
    // make a mapper 创建Mapper对象
    org.apache.hadoop.mapreduce.Mapper<INKEY,INVALUE,OUTKEY,OUTVALUE> mapper =
      (org.apache.hadoop.mapreduce.Mapper<INKEY,INVALUE,OUTKEY,OUTVALUE>)
        ReflectionUtils.newInstance(taskContext.getMapperClass(), job);
    // make the input format
    org.apache.hadoop.mapreduce.InputFormat<INKEY,INVALUE> inputFormat =
      (org.apache.hadoop.mapreduce.InputFormat<INKEY,INVALUE>)
        ReflectionUtils.newInstance(taskContext.getInputFormatClass(), job);
    // rebuild the input split
    org.apache.hadoop.mapreduce.InputSplit split = null;
    split = getSplitDetails(new Path(splitIndex.getSplitLocation()),
        splitIndex.getStartOffset());
    LOG.info("Processing split: " + split);

    org.apache.hadoop.mapreduce.RecordReader<INKEY,INVALUE> input =
      new NewTrackingRecordReader<INKEY,INVALUE>
        (split, inputFormat, reporter, taskContext);
    
    job.setBoolean(JobContext.SKIP_RECORDS, isSkipping());
    org.apache.hadoop.mapreduce.RecordWriter output = null;
    
    // get an output object
    if (job.getNumReduceTasks() == 0) {
      output = 
        new NewDirectOutputCollector(taskContext, job, umbilical, reporter);
    } else {
      output = new NewOutputCollector(taskContext, job, umbilical, reporter);
    }

    org.apache.hadoop.mapreduce.MapContext<INKEY, INVALUE, OUTKEY, OUTVALUE> 
    mapContext = 
      new MapContextImpl<INKEY, INVALUE, OUTKEY, OUTVALUE>(job, getTaskID(), 
          input, output, 
          committer, 
          reporter, split);

    org.apache.hadoop.mapreduce.Mapper<INKEY,INVALUE,OUTKEY,OUTVALUE>.Context 
        mapperContext = 
          new WrappedMapper<INKEY, INVALUE, OUTKEY, OUTVALUE>().getMapContext(
              mapContext);

    try {
      //初始化input对象  
      input.initialize(split, mapperContext);
      //运行mapper任务，调用自定义的Mapper类覆写的map方法  
      mapper.run(mapperContext);
      mapPhase.complete();
      //设置任务的当前阶段：排序  
      setPhase(TaskStatus.Phase.SORT);
      //更新任务的状态  
      statusUpdate(umbilical);
      //input对象的关闭与置空  
      input.close();
      input = null;
      //output对象的关闭与置空  
      output.close(mapperContext);
      output = null;
    } finally {
      closeQuietly(input);
      closeQuietly(output, mapperContext);
    }
 }

1.4.4 mapper、inputFormat和input对象的创建

1. Mapper对象如何创建的？

// make a mapper 创建Mapper对象
org.apache.hadoop.mapreduce.Mapper<INKEY,INVALUE,OUTKEY,OUTVALUE> mapper =
    (org.apache.hadoop.mapreduce.Mapper<INKEY,INVALUE,OUTKEY,OUTVALUE>)
    ReflectionUtils.newInstance(taskContext.getMapperClass(), job);

点击进入getMapperClass()方法中->JobContext接口中：

  public Class<? extends Mapper<?,?,?,?>> getMapperClass() 
     throws ClassNotFoundException;

选中getMapperClass()之后Ctrl+Alt+B->JobContextImpl:

public Class<? extends Mapper<?,?,?,?>> getMapperClass() 
     throws ClassNotFoundException {
    return (Class<? extends Mapper<?,?,?,?>>) 
      conf.getClass(MAP_CLASS_ATTR, Mapper.class);
  }

去配置文件中查找MAP_CLASS_ATTR常量对应（mapreduce.job.map.class）的自定义的Mapper类，如果找到则使用自定义的Mapper类，如果找不到使用Mapper.class。

使用mapreduce.job.map.class去job.xml中查找对应的Mapper类：

<property>
    <name>mapreduce.job.map.class</name>
    <value>com.itbaizhan.WCMapper</value>
    <final>false</final>
    <source>programmatically</source>
</property>

对应的自定义的Mapper类就是com.itbaizhan.WCMapper。对应的设置代码就是在WCDriver类中：

job.setMapperClass(WCMapper.class);
//也就是：

/**
   * Set the {@link Mapper} for the job.
   * @param cls the <code>Mapper</code> to use
   * @throws IllegalStateException if the job is submitted
   */
public void setMapperClass(Class<? extends Mapper> cls
                          ) throws IllegalStateException {
    ensureState(JobState.DEFINE);
    conf.setClass(MAP_CLASS_ATTR, cls, Mapper.class);
}

思考：为什么在WCDriver类设置好Mapper类之后可以在MapTask中直接使用？设置和使用谁先谁后？

参考答案：WCDriver在客户端执行的，搞定一切配置信息之后，才会提交作业，只有提交作业之后才开始运行MapTask.

2. inputFormat对象如何创建的？

 // make the input format
org.apache.hadoop.mapreduce.InputFormat<INKEY,INVALUE> inputFormat =
    (org.apache.hadoop.mapreduce.InputFormat<INKEY,INVALUE>)
    ReflectionUtils.newInstance(taskContext.getInputFormatClass(), job);

taskContext.getInputFormatClass()->JobContext接口中：

  public Class<? extends InputFormat<?,?>> getInputFormatClass() 
     throws ClassNotFoundException;

选择getInputFormatClass() 后Ctrl+Alt+B->选择JobContextImpl

public Class<? extends InputFormat<?,?>> getInputFormatClass() 
   throws ClassNotFoundException {
  return (Class<? extends InputFormat<?,?>>) 
    conf.getClass(INPUT_FORMAT_CLASS_ATTR, TextInputFormat.class);
}

如果设置过使用自定义的InputFormat类，没有指定就使用TextInputFormat。可以通过如下方式设置：

job.setInputFormatClass(KeyValueTextInputFormat.class);

默认使用的就是TextInputFormat类。

3. input对象如何创建的？

org.apache.hadoop.mapreduce.RecordReader<INKEY,INVALUE> input =
    new NewTrackingRecordReader<INKEY,INVALUE>
    (split, inputFormat, reporter, taskContext);

点击NewTrackingRecordReader<INKEY,INVALUE>()

NewTrackingRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                        org.apache.hadoop.mapreduce.InputFormat<K, V> inputFormat,
                        TaskReporter reporter,
                        org.apache.hadoop.mapreduce.TaskAttemptContext taskContext)
    throws InterruptedException, IOException {
    this.reporter = reporter;
    this.inputRecordCounter = reporter
        .getCounter(TaskCounter.MAP_INPUT_RECORDS);
    this.fileInputByteCounter = reporter
        .getCounter(FileInputFormatCounter.BYTES_READ);

    List <Statistics> matchedStats = null;
    if (split instanceof org.apache.hadoop.mapreduce.lib.input.FileSplit) {
        matchedStats = getFsStatistics(((org.apache.hadoop.mapreduce.lib.input.FileSplit) split)
                                       .getPath(), taskContext.getConfiguration());
    }
    fsStats = matchedStats;

    long bytesInPrev = getInputBytes(fsStats);
    //inputFormat默认是TextInputFormat对象创建的，createRecordReader()来至TextInputFormat
    this.real = inputFormat.createRecordReader(split, taskContext);
    long bytesInCurr = getInputBytes(fsStats);
    fileInputByteCounter.increment(bytesInCurr - bytesInPrev);
}

选择inputFormat.createRecordReader(split, taskContext)，点击：进入InputFormat中：

public abstract 
    RecordReader<K,V> createRecordReader(InputSplit split,
                                         TaskAttemptContext context
                                        ) throws IOException, InterruptedException;

TextInputFormat类的createRecordReader()：

  @Override
  public RecordReader<LongWritable, Text> 
    createRecordReader(InputSplit split,
                       TaskAttemptContext context) {
    String delimiter = context.getConfiguration().get(
        "textinputformat.record.delimiter");
    byte[] recordDelimiterBytes = null;
    if (null != delimiter)
      recordDelimiterBytes = delimiter.getBytes(Charsets.UTF_8);
    return new LineRecordReader(recordDelimiterBytes);
  }

最终实现的是LineRecordReader（行读取器）。

1.4.5 initialize方法源码分析

点击MapTask类的runNewMapper()方法的798行：

input.initialize(split, mapperContext);

进入到RecordReader抽象类中：

public abstract void initialize(InputSplit split,
                                  TaskAttemptContext context
                                  ) throws IOException, InterruptedException;

选中initialize方法，然后Ctrl+Alt+B->LineRecordReader类的initialize()方法中：

public void initialize(InputSplit genericSplit,
                         TaskAttemptContext context) throws IOException {
    FileSplit split = (FileSplit) genericSplit;
    Configuration job = context.getConfiguration();
    this.maxLineLength = job.getInt(MAX_LINE_LENGTH, Integer.MAX_VALUE);
    //起始偏移量=切片的起始偏移量
    start = split.getStart();
    //结束偏移量=起始偏移量+切片的大小  单位：B
    end = start + split.getLength();
    //获取切片对应文件的Path路径的对象
    final Path file = split.getPath();

    // open the file and seek to the start of the split
    //获取分布式文件系统对象
    final FileSystem fs = file.getFileSystem(job);
    //打开文件进行读取
    fileIn = fs.open(file);
    
    CompressionCodec codec = new CompressionCodecFactory(job).getCodec(file);
    if (null!=codec) {
      isCompressedInput = true;
      decompressor = CodecPool.getDecompressor(codec);
      if (codec instanceof SplittableCompressionCodec) {
        final SplitCompressionInputStream cIn =
          ((SplittableCompressionCodec)codec).createInputStream(
            fileIn, decompressor, start, end,
            SplittableCompressionCodec.READ_MODE.BYBLOCK);
        in = new CompressedSplitLineReader(cIn, job,
            this.recordDelimiterBytes);
        start = cIn.getAdjustedStart();
        end = cIn.getAdjustedEnd();
        filePosition = cIn;
      } else {
        //如果start不等于:说明当前MapTask处理的切片不是对应文件的第一个切片，存在行被切断的问题  
        if (start != 0) {
          // So we have a split that is only part of a file stored using
          // a Compression codec that cannot be split.
          throw new IOException("Cannot seek in " +
              codec.getClass().getSimpleName() + " compressed stream");
        }

        in = new SplitLineReader(codec.createInputStream(fileIn,
            decompressor), job, this.recordDelimiterBytes);
        filePosition = fileIn;
      }
    } else {//当前MapTask处理是对应文件的第一个切片，不存在行被切断的风险
      fileIn.seek(start);
      in = new UncompressedSplitLineReader(
          fileIn, job, this.recordDelimiterBytes, split.getLength());
      filePosition = fileIn;
    }
    // If this is not the first split, we always throw away first record
    // because we always (except the last split) read one extra line in
    // next() method.
    //如果start!=0，第一行不处理，
    if (start != 0) {
      start += in.readLine(new Text(), 0, maxBytesToConsume(start));
    }
    this.pos = start;
  }

从第二个切片开始，都放弃第一行的处理，那么Map任务处理的时候，就不存在断行的问题了。每个切片对应的Map任务都会向后多读取一行并进行处理。

1.4.6 run(mapperContext)

点击MapTask类的runNewMapper()方法的799行：

mapper.run(mapperContext);

Ctrl+单击run方法，进入Mapper类的run方法中：

public void run(Context context) throws IOException, InterruptedException {
    //自定义的Mapper类也可以覆写setup(context)和cleanup(context)方法，
    //一个Map任务这两个方法都会执行一次。
    setup(context);
    try {
      while (context.nextKeyValue()) {//是否还有未处理的内容
        //调用自定义的Mapper类(比如WCMapper类)中的map方法  
        map(context.getCurrentKey(), context.getCurrentValue(), context);
      }
    } finally {
      cleanup(context);
    }
  }

1.4.7 nextKeyValue()方法：

点击context.nextKeyValue()方法，进入到接口TaskInputOutputContext中：

public boolean nextKeyValue() throws IOException, InterruptedException;

选中方法nextKeyValue()->Ctrl+Alt+B->MapContextImpl类的nextKeyValue()方法：

  @Override
  public boolean nextKeyValue() throws IOException, InterruptedException {
    return reader.nextKeyValue();
  }

点击：reader.nextKeyValue()->RecodeReader类中：

public abstract 
  boolean nextKeyValue() throws IOException, InterruptedException;

选中nextKeyValue()->LineRecordReader的nextKeyValue()方法：

//判断是否还存在未处理的键值对
//LineRecordReader类的注释：Treats keys as offset in file and value as line
//key是偏移量
//value：当前行的内容
public boolean nextKeyValue() throws IOException {
    if (key == null) {
        key = new LongWritable();
    }
    key.set(pos);
    if (value == null) {
        value = new Text();
    }
    int newSize = 0;
    // We always read one extra line, which lies outside the upper
    // split limit i.e. (end - 1)
    while (getFilePosition() <= end || in.needAdditionalRecordAfterSplit()) {
        if (pos == 0) {
            newSize = skipUtfByteOrderMark();
        } else {
            //读取当前行的内容赋值给value
            newSize = in.readLine(value, maxLineLength, maxBytesToConsume(pos));
            //将偏移量改变
            pos += newSize;
        }

        if ((newSize == 0) || (newSize < maxLineLength)) {
            break;
        }

        // line too long. try again
        LOG.info("Skipped line of size " + newSize + " at pos " + 
                 (pos - newSize));
    }
    if (newSize == 0) {
        key = null;
        value = null;
        return false;
    } else {
        return true;
    }
}

LineRecordReader类的中getCurrentKey()和getCurrentValue():

  @Override
  public LongWritable getCurrentKey() {
    return key;
  }

  @Override
  public Text getCurrentValue() {
    return value;
  }

接着调用自定义Mapper类中的map方法：

//文本中的每一行内容调用一次map方法
@Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {
    Thread.sleep(99999999);
    if(value!=null) {
        //value就是读取到的当前行中的内容,并将之转换为字符串
        String lineContenct = value.toString();
        //将字符串进行处理：先去掉两端的空格，然后在按照空格进行切分
        String[] words = lineContenct.trim().split(" ");
        //遍历数组，逐一进行输出
        for(String word:words){
            //将word内容封装到keyOut中
            keyOut.set(word);
            //将kv对输出
            context.write(keyOut, valueOut);
        }
    }
}

在这里插入图片描述

1.4.8 out对象如何创建

在这里插入图片描述

自定义的WCMapper类的map方法，将kv对输出时使用：context.write(keyOut, valueOut);

context如何来的？从run(Context context)传递过来的，也就是：mapper.run(mapperContext)

接下来分析mapperContext从何而来？

MapTask类的runNewMapper方法中：

在这里插入图片描述

设置输出：如果作业只有map任务没有reduce任务，则直接通过NewDirectOutputCollector写到HDFS上。如果有reduce任务，通过NewOutputCollector写到圆形缓冲区中。

1.4.9 有Reduce时out对象创建

通过NewOutputCollector写到圆形缓冲区中。

NewOutputCollector(org.apache.hadoop.mapreduce.JobContext jobContext,
                   JobConf job,
                   TaskUmbilicalProtocol umbilical,
                   TaskReporter reporter
                  ) throws IOException, ClassNotFoundException {
    //涉及到后续的圆形缓冲区，稍后讲：
    collector = createSortingCollector(job, reporter);
    //获取reducer任务的数量，作为分区数量，所以分区和reduce任务一对一。
    partitions = jobContext.getNumReduceTasks();
    if (partitions > 1) {//有2个或2个以上的分区
        //创建分区对象
        partitioner = (org.apache.hadoop.mapreduce.Partitioner<K,V>)
            ReflectionUtils.newInstance(jobContext.getPartitionerClass(), job);
    } else {//=1，直接返回 0
        partitioner = new org.apache.hadoop.mapreduce.Partitioner<K,V>() {
            @Override
            public int getPartition(K key, V value, int numPartitions) {
                return partitions - 1;
            }
        };
    }
}

1.4.10 分区

如果reduce任务数量大于1，也就是分区数量大于1，调用jobContext.getPartitionerClass()来获取分区类的Class对象。单击getPartitionerClass()->JobContext接口中：

public Class<? extends Partitioner<?,?>> getPartitionerClass() 
     throws ClassNotFoundException;

Ctrl+Alt+B->JobContextImpl:

 public Class<? extends Partitioner<?,?>> getPartitionerClass() 
     throws ClassNotFoundException {
    return (Class<? extends Partitioner<?,?>>) 
      conf.getClass(PARTITIONER_CLASS_ATTR, HashPartitioner.class);
  }

获取PARTITIONER_CLASS_ATTR常量的值mapreduce.job.partitioner.class,该字符串去job.xml找它对应的value值。如果没有定义，则使用默认的HashPartitioner类作为分区类。

/** Partition keys by their {@link Object#hashCode()}. */
@InterfaceAudience.Public
@InterfaceStability.Stable
public class HashPartitioner<K, V> extends Partitioner<K, V> {

  /** Use {@link Object#hashCode()} to partition. */
  public int getPartition(K key, V value,
                          int numReduceTasks) {
    return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
  }
}

默认的分区的逻辑是，使用key的hashCode值%reduce任务数(也是分区总数) 求余数。

如何设置自定义的分区类：

job.setPartitionerClass(WCPartitioner.class);
job.setNumReduceTasks(4);

setPartitionerClass()方法的代码：

  public void setPartitionerClass(Class<? extends Partitioner> cls
                                  ) throws IllegalStateException {
    ensureState(JobState.DEFINE);
    conf.setClass(PARTITIONER_CLASS_ATTR, cls, 
                  Partitioner.class);
  }

自定义分区类如何定义：

//自定义分区类的key，value的类型，要分别对应Mapper输出的key和value的类型
//自定义的分区列一定要继承Partitioner类，并覆写getPartition方法
//分区的原则：避免数据倾斜的出现
public class WCPartitioner extends Partitioner<Text, IntWritable> {
    //abcdefghi   jklmnopqr  stuvwzyz 其他的
    @Override
    public int getPartition(Text key, IntWritable value, int numPartitions) {
        //key转换为字符串
        String word = key.toString();
        char ch = word.charAt(0);
        if(ch>='a'&&ch<='i'){
            return 0;
        }else if(ch>='j'&&ch<='r'){
            return 1;
        }else if(ch>='s'&&ch<='z'){
            return 2;
        }else{
            return 3;
        }
    }
}

1.4.11 圆形缓冲区引入

如果有reduce任务，通过NewOutputCollector写到圆形缓冲区中。所以自定义Mapper类在执行context.write(xx)方法时调用的就是NewOutputCollector类中的write方法：

@Override
public void write(K key, V value) throws IOException, InterruptedException {
  //将(key,value,partitin)缩写(k,v,p)
  collector.collect(key, value,
                    partitioner.getPartition(key, value, partitions));
}

放回到构造方法中：

NewOutputCollector(org.apache.hadoop.mapreduce.JobContext jobContext,
                   JobConf job,
                   TaskUmbilicalProtocol umbilical,
                   TaskReporter reporter
                  ) throws IOException, ClassNotFoundException {
    //涉及到后续的圆形缓冲区，稍后讲：
    collector = createSortingCollector(job, reporter);
    ......
}

点击createSortingCollector方法：

在这里插入图片描述

所以该collector就是找MAP_OUTPUT_COLLECTOR_CLASS_ATTR对应的配置的类，如果没有配置则使用默认的MapOutputBuffer类来创建对象。MapOutputBuffer就是圆形缓冲区的类。

1.4.12 collector.init方法

在这里插入图片描述

点击MapTask类中的createSortingCollector()->单击collector.init(context);

public interface MapOutputCollector<K, V> {
  public void init(Context context
                  ) throws IOException, ClassNotFoundException;
}

选中init方法，Ctrl+Alt+B->选择MapOutputBuffer类：

public void init(MapOutputCollector.Context context
                    ) throws IOException, ClassNotFoundException {
      job = context.getJobConf();
      reporter = context.getReporter();
      mapTask = context.getMapTask();
      mapOutputFile = mapTask.getMapOutputFile();
      sortPhase = mapTask.getSortPhase();
      spilledRecordsCounter = reporter.getCounter(TaskCounter.SPILLED_RECORDS);
      //获取分区的数量也就是reduce的数量
      partitions = job.getNumReduceTasks();
      rfs = ((LocalFileSystem)FileSystem.getLocal(job)).getRaw();

      //sanity checks  阈值默认 0.8
      //MAP_SORT_SPILL_PERCENT = "mapreduce.map.sort.spill.percent"
      final float spillper =
        job.getFloat(JobContext.MAP_SORT_SPILL_PERCENT, (float)0.8);
      //圆形缓冲区的大小100MB
      //IO_SORT_MB = "mapreduce.task.io.sort.mb"
      //DEFAULT_IO_SORT_MB = 100;
      final int sortmb = job.getInt(MRJobConfig.IO_SORT_MB,
          MRJobConfig.DEFAULT_IO_SORT_MB);
      indexCacheMemoryLimit = job.getInt(JobContext.INDEX_CACHE_MEMORY_LIMIT,
                                         INDEX_CACHE_MEMORY_LIMIT_DEFAULT);
      if (spillper > (float)1.0 || spillper <= (float)0.0) {
        throw new IOException("Invalid \"" + JobContext.MAP_SORT_SPILL_PERCENT +
            "\": " + spillper);
      }
      if ((sortmb & 0x7FF) != sortmb) {
        throw new IOException(
            "Invalid \"" + JobContext.IO_SORT_MB + "\": " + sortmb);
      }
      //MAP_SORT_CLASS = "map.sort.class" 如果配置了该参数对应的类，则使用它进行排序
      //如果没有设置过，使用默认的QuickSort（快排）
      sorter = ReflectionUtils.newInstance(job.getClass(
                   MRJobConfig.MAP_SORT_CLASS, QuickSort.class,
                   IndexedSorter.class), job);
      // buffers and accounting  100变为二进制，然后向左位移20，
      int maxMemUsage = sortmb << 20;
      maxMemUsage -= maxMemUsage % METASIZE;
      //定义一个字节数组，存放圆形缓冲区中的（k,v,p）对
      //圆形缓冲区本质上是默认长度为100MB一个字节数组，通过一定的算法实现逻辑上的圆形缓存区的效果。
      kvbuffer = new byte[maxMemUsage];
      bufvoid = kvbuffer.length;
      kvmeta = ByteBuffer.wrap(kvbuffer)
         .order(ByteOrder.nativeOrder())
         .asIntBuffer();
      setEquator(0);
      bufstart = bufend = bufindex = equator;
      kvstart = kvend = kvindex;

      maxRec = kvmeta.capacity() / NMETA;
      softLimit = (int)(kvbuffer.length * spillper);
      bufferRemaining = softLimit;
      LOG.info(JobContext.IO_SORT_MB + ": " + sortmb);
      LOG.info("soft limit at " + softLimit);
      LOG.info("bufstart = " + bufstart + "; bufvoid = " + bufvoid);
      LOG.info("kvstart = " + kvstart + "; length = " + maxRec);

      // k/v serialization 排序比较器
      comparator = job.getOutputKeyComparator();
      //mapper类输出key的类型
      keyClass = (Class<K>)job.getMapOutputKeyClass();
      //mapper类输出value的类型
      valClass = (Class<V>)job.getMapOutputValueClass();
      serializationFactory = new SerializationFactory(job);
      keySerializer = serializationFactory.getSerializer(keyClass);
      keySerializer.open(bb);
      valSerializer = serializationFactory.getSerializer(valClass);
      valSerializer.open(bb);

      // output counters 计算器  字节数、记录数等
      mapOutputByteCounter = reporter.getCounter(TaskCounter.MAP_OUTPUT_BYTES);
      mapOutputRecordCounter =
        reporter.getCounter(TaskCounter.MAP_OUTPUT_RECORDS);
      fileOutputByteCounter = reporter
          .getCounter(TaskCounter.MAP_OUTPUT_MATERIALIZED_BYTES);

      // compression  压缩处理
      if (job.getCompressMapOutput()) {
        Class<? extends CompressionCodec> codecClass =
          job.getMapOutputCompressorClass(DefaultCodec.class);
        codec = ReflectionUtils.newInstance(codecClass, job);
      } else {
        codec = null;
      }

      // combiner 类相关
      final Counters.Counter combineInputCounter =
        reporter.getCounter(TaskCounter.COMBINE_INPUT_RECORDS);
      combinerRunner = CombinerRunner.create(job, getTaskID(), 
                                             combineInputCounter,
                                             reporter, null);
      if (combinerRunner != null) {
        final Counters.Counter combineOutputCounter =
          reporter.getCounter(TaskCounter.COMBINE_OUTPUT_RECORDS);
        combineCollector= new CombineOutputCollector<K,V>(combineOutputCounter, reporter, job);
      } else {
        combineCollector = null;
      }
      spillInProgress = false;
      //MAP_COMBINE_MIN_SPILLS = "mapreduce.map.combine.minspills"
      //如果配置文件中配置了就使用配置的值，没有配置默认为3个。
      minSpillsForCombine = job.getInt(JobContext.MAP_COMBINE_MIN_SPILLS, 3);
      //启动一个名称为SpillThread守护线程，进行写溢出文件
      spillThread.setDaemon(true);
      spillThread.setName("SpillThread");
      spillLock.lock();
      try {
        //启动线程
        spillThread.start();
        while (!spillThreadRunning) {
          spillDone.await();
        }
      } catch (InterruptedException e) {
        throw new IOException("Spill thread failed to initialize", e);
      } finally {
        spillLock.unlock();
      }
      if (sortSpillException != null) {
        throw new IOException("Spill thread failed to initialize",
            sortSpillException);
      }
 }

1.4.13 排序比较器

点击MapTask类中的createSortingCollector()->单击collector.init(context);

public interface MapOutputCollector<K, V> {
  public void init(Context context
                  ) throws IOException, ClassNotFoundException;
}

选中init方法，Ctrl+Alt+B->选择MapOutputBuffer类1018行：

comparator = job.getOutputKeyComparator();

点击getOutputKeyComparator()方法：

  public RawComparator getOutputKeyComparator() {
    //获取KEY_COMPARATOR对应排序比较器，如果有获取出来，没有的话返回null  
    Class<? extends RawComparator> theClass = getClass(
      JobContext.KEY_COMPARATOR, null, RawComparator.class);
    //如果设置过排序比较器，则通过反射创建对应类的对象  
    if (theClass != null)
      return ReflectionUtils.newInstance(theClass, this);
    //如果没有设置过，通过如下方式获取对应key的类中的自带的一个比较器，比如Text类。  
    return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class), this);
  }

如何设置排序比较器？

job.setSortComparatorClass(WCSortComparator.class);

点击如上方法：

public void setSortComparatorClass(Class<? extends RawComparator> cls
                                  ) throws IllegalStateException {
    ensureState(JobState.DEFINE);
    conf.setOutputKeyComparatorClass(cls);
}

点击setOutputKeyComparatorClass(cls)方法：

public void setOutputKeyComparatorClass(Class<? extends RawComparator> theClass) {
    setClass(JobContext.KEY_COMPARATOR,
             theClass, RawComparator.class);
}

JobContext.KEY_COMPARATOR就是获取是使用到的常量，来找它对应的value。

如果没有设置排序比较器，如何从Key对应的类中找比较器？

常见的Key对应的类如Text、LongWritable等。

Ctrl+Shift+R->Text.java

//在Text类内部定义的比较器的类Comparator
//1.继承WritableComparator抽象类
public static class Comparator extends WritableComparator {
    //2.添加一个无参数的构造方法
    public Comparator() {
      //3.通过super(Text.class) 调用父类的构造方法，并传递外部类Text.class  
      super(Text.class);
    }
	/*4.根据需要覆写父类的比较的方法
	也可以覆写其他的方法，比如：
	public int compare(WritableComparable a, WritableComparable b) {
        return a.compareTo(b);
    }
	*/
    @Override
    public int compare(byte[] b1, int s1, int l1,
                       byte[] b2, int s2, int l2) {
      int n1 = WritableUtils.decodeVIntSize(b1[s1]);
      int n2 = WritableUtils.decodeVIntSize(b2[s2]);
      return compareBytes(b1, s1+n1, l1-n1, b2, s2+n2, l2-n2);
    }
  }
//5.注册比较器的类
static {
    // register this comparator
    WritableComparator.define(Text.class, new Comparator());
}

Ctrl+点击如上的define(),查看如何注册的比较器。

private static final ConcurrentHashMap<Class, WritableComparator> comparators 
          = new ConcurrentHashMap<Class, WritableComparator>();
public static void define(Class c, WritableComparator comparator) {
    comparators.put(c, comparator);
}

读取Key对应类注册的比较器的实例:

WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class),

Ctrl+单击如上的get方法

public static WritableComparator get(
      Class<? extends WritableComparable> c, Configuration conf) {
    WritableComparator comparator = comparators.get(c);
    if (comparator == null) {
      // force the static initializers to run
      forceInit(c);
      // look to see if it is defined now
      comparator = comparators.get(c);
      // if not, use the generic one
      if (comparator == null) {
        comparator = new WritableComparator(c, conf, true);
      }
    }
    // Newly passed Configuration objects should be used.
    ReflectionUtils.setConf(comparator, conf);
    return comparator;
  }

就从我们注册是存入的Map中获取的：comparators.get©。

comparator在哪里使用的？

Ctrl+Shift+R->SpillThread->run()方法->sortAndSpill()：

sorter.sort(MapOutputBuffer.this, mstart, mend, reporter)

Ctrl+单击 sort()

void sort(IndexedSortable s, int l, int r, Progressable rep);

Ctrl+Alt+B->QuickSort类：

  public void sort(final IndexedSortable s, int p, int r,
      final Progressable rep) {
    sortInternal(s, p, r, rep, getMaxDepth(r - p));
  }
private static void sortInternal(final IndexedSortable s, int p, int r,
      final Progressable rep, int depth) {
    if (null != rep) {
      rep.progress();
    }
    while (true) {
    if (r-p < 13) {
      for (int i = p; i < r; ++i) {
        for (int j = i; j > p && s.compare(j-1, j) > 0; --j) {
          s.swap(j, j-1);
        }
      }
      return;
    }
    ......
}

compare()->IndexedSortable接口的compare()方法->Ctrl+Alt+B->MapOutputBuffer类的compare方法：

public int compare(final int mi, final int mj) {
  final int kvi = offsetFor(mi % maxRec);
  final int kvj = offsetFor(mj % maxRec);
  final int kvip = kvmeta.get(kvi + PARTITION);
  final int kvjp = kvmeta.get(kvj + PARTITION);
  // sort by partition
  if (kvip != kvjp) {
    return kvip - kvjp;
  }
  // sort by key 就使用到了comparator比较器对象
  return comparator.compare(kvbuffer,
      kvmeta.get(kvi + KEYSTART),
      kvmeta.get(kvi + VALSTART) - kvmeta.get(kvi + KEYSTART),
      kvbuffer,
      kvmeta.get(kvj + KEYSTART),
      kvmeta.get(kvj + VALSTART) - kvmeta.get(kvj + KEYSTART));
}

1.4.14 没有Reduce时out对象创建

直接通过NewDirectOutputCollector写到HDFS上

MapTask类的runNewMapper方法中：

在这里插入图片描述

MapTask->NewDirectOutputCollector->write()方法：

public void write(K key, V value) 
    throws IOException, InterruptedException {
    reporter.progress();
    long bytesOutPrev = getOutputBytes(fsStats);
    out.write(key, value);
    long bytesOutCurr = getOutputBytes(fsStats);
    fileOutputByteCounter.increment(bytesOutCurr - bytesOutPrev);
    mapOutputRecordCounter.increment(1);
}

分析out对象来源？

NewDirectOutputCollector(MRJobConfig jobContext,
        JobConf job, TaskUmbilicalProtocol umbilical, TaskReporter reporter) 
    throws IOException, ClassNotFoundException, InterruptedException {
      this.reporter = reporter;
      mapOutputRecordCounter = reporter
          .getCounter(TaskCounter.MAP_OUTPUT_RECORDS);
      fileOutputByteCounter = reporter
          .getCounter(FileOutputFormatCounter.BYTES_WRITTEN);

      List<Statistics> matchedStats = null;
      if (outputFormat instanceof org.apache.hadoop.mapreduce.lib.output.FileOutputFormat) {
        matchedStats = getFsStatistics(org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
            .getOutputPath(taskContext), taskContext.getConfiguration());
      }
      fsStats = matchedStats;

      long bytesOutPrev = getOutputBytes(fsStats);
      out = outputFormat.getRecordWriter(taskContext);
      long bytesOutCurr = getOutputBytes(fsStats);
      fileOutputByteCounter.increment(bytesOutCurr - bytesOutPrev);
    }

点击getRecordWriter()->Ctrl+Alt+B->MapFileOutputFormat

public RecordWriter<WritableComparable<?>, Writable> getRecordWriter(
      TaskAttemptContext context) throws IOException {
    Configuration conf = context.getConfiguration();
    CompressionCodec codec = null;
    CompressionType compressionType = CompressionType.NONE;
    if (getCompressOutput(context)) {
      // find the kind of compression to do
      compressionType = SequenceFileOutputFormat.getOutputCompressionType(context);

      // find the right codec
      Class<?> codecClass = getOutputCompressorClass(context,
	                          DefaultCodec.class);
      codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, conf);
    }

    Path file = getDefaultWorkFile(context, "");
    //将内容直接写入HDFS文件系统中
    FileSystem fs = file.getFileSystem(conf);
    // ignore the progress parameter, since MapFile is local
    final MapFile.Writer out =
      new MapFile.Writer(conf, fs, file.toString(),
        context.getOutputKeyClass().asSubclass(WritableComparable.class),
        context.getOutputValueClass().asSubclass(Writable.class),
        compressionType, codec, context);

    return new RecordWriter<WritableComparable<?>, Writable>() {
        public void write(WritableComparable<?> key, Writable value)
            throws IOException {
          out.append(key, value);
        }

        public void close(TaskAttemptContext context) throws IOException { 
          out.close();
        }
      };
  }

1.5 Reduce阶段源码分析

1.5.1 概述

在这里插入图片描述

阅读源码需要解决以下问题：

默认启动5个线程去完成MapTask任务的节点上分别抓取数据。
通过什么方式去Map端抓取数据的http或https get？
如何进行分组的？
reduce端的二次排序是如何进行的？

分析ReduceTask类的run方法：

public void run(JobConf job, final TaskUmbilicalProtocol umbilical)
    throws IOException, InterruptedException, ClassNotFoundException {
    job.setBoolean(JobContext.SKIP_RECORDS, isSkipping());

    if (isMapOrReduce()) {
      //copy：从完成MapTask的Map端的节点中拷贝处理过的数据  
      copyPhase = getProgress().addPhase("copy");
      //排序  
      sortPhase  = getProgress().addPhase("sort");
      //reduce处理  
      reducePhase = getProgress().addPhase("reduce");
    }
    // start thread that will handle communication with parent
    TaskReporter reporter = startReporter(umbilical);
    boolean useNewApi = job.getUseNewReducer();
    //初始化处理
    initialize(job, getJobID(), reporter, useNewApi);

    // check if it is a cleanupJobTask
    if (jobCleanup) {
      runJobCleanupTask(umbilical, reporter);
      return;
    }
    if (jobSetup) {
      runJobSetupTask(umbilical, reporter);
      return;
    }
    if (taskCleanup) {
      runTaskCleanupTask(umbilical, reporter);
      return;
    }
    
    // Initialize the codec
    codec = initCodec();
    RawKeyValueIterator rIter = null;
    ShuffleConsumerPlugin shuffleConsumerPlugin = null;
    //combinerClass获取
    Class combinerClass = conf.getCombinerClass();
    CombineOutputCollector combineCollector = 
      (null != combinerClass) ? 
     new CombineOutputCollector(reduceCombineOutputCounter, reporter, conf) : null;

    Class<? extends ShuffleConsumerPlugin> clazz =
          job.getClass(MRConfig.SHUFFLE_CONSUMER_PLUGIN, Shuffle.class, ShuffleConsumerPlugin.class);
					
    shuffleConsumerPlugin = ReflectionUtils.newInstance(clazz, job);
    LOG.info("Using ShuffleConsumerPlugin: " + shuffleConsumerPlugin);

    ShuffleConsumerPlugin.Context shuffleContext = 
      new ShuffleConsumerPlugin.Context(getTaskID(), job, FileSystem.getLocal(job), umbilical, 
                  super.lDirAlloc, reporter, codec, 
                  combinerClass, combineCollector, 
                  spilledRecordsCounter, reduceCombineInputCounter,
                  shuffledMapsCounter,
                  reduceShuffleBytes, failedShuffleCounter,
                  mergedMapOutputsCounter,
                  taskStatus, copyPhase, sortPhase, this,
                  mapOutputFile, localMapFiles);
    //Shuffle插件初始化
    shuffleConsumerPlugin.init(shuffleContext);
    //Shuffle插件执行
    rIter = shuffleConsumerPlugin.run();

    // free up the data structures
    mapOutputFilesOnDisk.clear();
    
    sortPhase.complete();                         // sort is complete
    setPhase(TaskStatus.Phase.REDUCE); 
    statusUpdate(umbilical);
    Class keyClass = job.getMapOutputKeyClass();
    Class valueClass = job.getMapOutputValueClass();
    //分组比较器
    RawComparator comparator = job.getOutputValueGroupingComparator();

    if (useNewApi) {//hadoop2.x+ 使用
      runNewReducer(job, umbilical, reporter, rIter, comparator, 
                    keyClass, valueClass);
    } else {//hadoop1.x 使用
      runOldReducer(job, umbilical, reporter, rIter, comparator, 
                    keyClass, valueClass);
    }

    shuffleConsumerPlugin.close();
    done(umbilical, reporter);
  }

1.5.2 copy数据源码

入口ReduceTask类的run方法中：

//Shuffle插件初始化
shuffleConsumerPlugin.init(shuffleContext);
//Shuffle插件执行
rIter = shuffleConsumerPlugin.run();

init方法：

public interface ShuffleConsumerPlugin<K, V> {
  public void init(Context<K, V> context);
 }

Ctrl+Alt+B->Shuffle

@Override
  public void init(ShuffleConsumerPlugin.Context context) {
    this.context = context;
	//从context对象中获取需要的值
    this.reduceId = context.getReduceId();
    this.jobConf = context.getJobConf();
    this.umbilical = context.getUmbilical();
    this.reporter = context.getReporter();
    this.metrics = ShuffleClientMetrics.create();
    this.copyPhase = context.getCopyPhase();
    this.taskStatus = context.getStatus();
    this.reduceTask = context.getReduceTask();
    this.localMapFiles = context.getLocalMapFiles();
    //实例化对象
    scheduler = new ShuffleSchedulerImpl<K, V>(jobConf, taskStatus, reduceId,
        this, copyPhase, context.getShuffledMapsCounter(),
        context.getReduceShuffleBytes(), context.getFailedShuffleCounter());
    //创建merger对象  
    merger = createMergeManager(context);
  }

run()

public RawKeyValueIterator run() throws IOException, InterruptedException;

选中run()，Ctrl+Alt+B->Shuffle类：

@Override
  public RawKeyValueIterator run() throws IOException, InterruptedException {
    // Scale the maximum events we fetch per RPC call to mitigate OOM issues
    // on the ApplicationMaster when a thundering herd of reducers fetch events
    // TODO: This should not be necessary after HADOOP-8942
    int eventsPerReducer = Math.max(MIN_EVENTS_TO_FETCH,
        MAX_RPC_OUTSTANDING_EVENTS / jobConf.getNumReduceTasks());
    int maxEventsToFetch = Math.min(MAX_EVENTS_TO_FETCH, eventsPerReducer);

    // Start the map-completion events fetcher thread
    final EventFetcher<K,V> eventFetcher = 
      new EventFetcher<K,V>(reduceId, umbilical, scheduler, this,
          maxEventsToFetch);
    eventFetcher.start();
    
    // Start the map-output fetcher threads  判断是否为本地运行
    boolean isLocal = localMapFiles != null;
    //如果本地运行（MapTask和ReduceTask在同一个节点中），仅需一个线程进行copy
    //如果yarn运行，首先获取SHUFFLE_PARALLEL_COPIES对应配置的值，未配置默认为5个  
    final int numFetchers = isLocal ? 1 :
      jobConf.getInt(MRJobConfig.SHUFFLE_PARALLEL_COPIES, 5);
    //创建1、5（或者自定义）个Fetcher对象。 
    Fetcher<K,V>[] fetchers = new Fetcher[numFetchers];
    if (isLocal) {//仅需启动一个线程即可
      fetchers[0] = new LocalFetcher<K, V>(jobConf, reduceId, scheduler,
          merger, reporter, metrics, this, reduceTask.getShuffleSecret(),
          localMapFiles);
      fetchers[0].start();
    } else {
      //numFetchers：为自定义的数量或默认值5
      for (int i=0; i < numFetchers; ++i) {
        //实例化对象，并分别启动一个对应的线程  
        fetchers[i] = new Fetcher<K,V>(jobConf, reduceId, scheduler, merger, 
                                       reporter, metrics, this, 
                                       reduceTask.getShuffleSecret());
        fetchers[i].start();//copy文件的操作就在Fetcher类run方法中。
      }
    }
    
    // Wait for shuffle to complete successfully
    while (!scheduler.waitUntilDone(PROGRESS_FREQUENCY)) {
      reporter.progress();
      
      synchronized (this) {
        if (throwable != null) {
          throw new ShuffleError("error in shuffle in " + throwingThreadName,
                                 throwable);
        }
      }
    }

    // Stop the event-fetcher thread
    eventFetcher.shutDown();
    
    // Stop the map-output fetcher threads
    for (Fetcher<K,V> fetcher : fetchers) {
      fetcher.shutDown();
    }
    
    // stop the scheduler
    scheduler.close();

    copyPhase.complete(); // copy is already complete
    taskStatus.setPhase(TaskStatus.Phase.SORT);
    reduceTask.statusUpdate(umbilical);

    // Finish the on-going merges...
    RawKeyValueIterator kvIter = null;
    try {
      kvIter = merger.close();
    } catch (Throwable e) {
      throw new ShuffleError("Error while doing final merge " , e);
    }

    // Sanity check
    synchronized (this) {
      if (throwable != null) {
        throw new ShuffleError("error in shuffle in " + throwingThreadName,
                               throwable);
      }
    }
    return kvIter;
  }

Fetcher类的构造方法：

setName("fetcher#" + id);
setDaemon(true);//后台线程

Fetcher类run方法：

public void run() {
    try {
      while (!stopped && !Thread.currentThread().isInterrupted()) {
        MapHost host = null;
        try {
          // If merge is on, block
          merger.waitForResource();

          // Get a host to shuffle from  获取完成MapTask的节点的Host
          host = scheduler.getHost();
          metrics.threadBusy();

          // Shuffle 从Host对应的节点中copy数据
          copyFromHost(host);
        } finally {
          if (host != null) {
            scheduler.freeHost(host);
            metrics.threadFree();            
          }
        }
      }
    } catch (InterruptedException ie) {
      return;
    } catch (Throwable t) {
      exceptionReporter.reportException(t);
    }
  }

copyFromHost(host)：

protected void copyFromHost(MapHost host) throws IOException {
    .....
        // Construct the url and connect
        URL url = getMapOutputURL(host, maps);
    DataInputStream input = null;
    try {
        input = openShuffleUrl(host, remaining, url);
        if (input == null) {
            return;
        }
        ......
    }
}

openShuffleUrl(host, remaining, url)：

private DataInputStream openShuffleUrl(MapHost host,
                                       Set<TaskAttemptID> remaining, URL url) {
    DataInputStream input = null;
    try {
        //进入该方法查看
        setupConnectionsWithRetry(url);
        if (stopped) {
            abortConnect(host, remaining);
        } else {
            //connection代表一个http或https连接
            input = new DataInputStream(connection.getInputStream());
        }
    } catch (TryAgainLaterException te) {
        LOG.warn("Connection rejected by the host " + te.host +
                 ". Will retry later.");
        scheduler.penalize(host, te.backoff);
    } catch (IOException ie) {
        boolean connectExcpt = ie instanceof ConnectException;
        ioErrs.increment(1);
        LOG.warn("Failed to connect to " + host + " with " + remaining.size() +
                 " map outputs", ie);

        // If connect did not succeed, just mark all the maps as failed,
        // indirectly penalizing the host
        scheduler.hostFailed(host.getHostName());
        for(TaskAttemptID left: remaining) {
            scheduler.copyFailed(left, host, false, connectExcpt);
        }
    }

    return input;
}

setupConnectionsWithRetry():

private void setupConnectionsWithRetry(URL url) throws IOException {
    openConnectionWithRetry(url);
    if (stopped) {
      return;
    }
      
    // generate hash of the url
    String msgToEncode = SecureShuffleUtils.buildMsgFrom(url);
    String encHash = SecureShuffleUtils.hashFromString(msgToEncode,
        shuffleSecretKey);
    
    setupShuffleConnection(encHash);
    connect(connection, connectionTimeout);
    // verify that the thread wasn't stopped during calls to connect
    if (stopped) {
      return;
    }
    
    verifyConnection(url, msgToEncode, encHash);
  }

openConnectionWithRetry():

private void openConnectionWithRetry(URL url) throws IOException {
    long startTime = Time.monotonicNow();
    boolean shouldWait = true;
    while (shouldWait) {
        try {
            //打开连接
            openConnection(url);
            shouldWait = false;
        }
        ......
    }
}

openConnection(url)：

@VisibleForTesting
  protected synchronized void openConnection(URL url)
      throws IOException {
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();
    if (sslShuffle) {
      HttpsURLConnection httpsConn = (HttpsURLConnection) conn;
      try {
        httpsConn.setSSLSocketFactory(sslFactory.createSSLSocketFactory());
      } catch (GeneralSecurityException ex) {
        throw new IOException(ex);
      }
      httpsConn.setHostnameVerifier(sslFactory.getHostnameVerifier());
    }
    connection = conn;
  }

在reduce端默认启动5个守护线程进行数据的copy，copy数据是通过http get的方式进行的。

1.5.3 分组比较器

ReduceTask类的run方法：

//获取分组比较器
RawComparator comparator = job.getOutputValueGroupingComparator();

getOutputValueGroupingComparator():

public RawComparator getOutputValueGroupingComparator() {
    //首先读取自定义的分组比较器
    Class<? extends RawComparator> theClass = getClass(
        JobContext.GROUP_COMPARATOR_CLASS, null, RawComparator.class);
    if (theClass == null) {
        //如果没有定义分组比较器，获取比起器
        return getOutputKeyComparator();
    }

    return ReflectionUtils.newInstance(theClass, this);
}

先获取指定的排序比较器，如果没有自定义排序比较器，则获取key对应类中注册的比较器。

public RawComparator getOutputKeyComparator() {
    Class<? extends RawComparator> theClass = getClass(
      JobContext.KEY_COMPARATOR, null, RawComparator.class);
    if (theClass != null)
      return ReflectionUtils.newInstance(theClass, this);
    return WritableComparator.get(getMapOutputKeyClass().asSubclass(WritableComparable.class), this);
  }

比较器类型	MapTask	ReduceTask
分组比较器		用户自定义
排序比较	用户自定义	用户自定义
自带比较器	key自带	key自带

MapTask&ReduceTask 使用Key对应类注册的比较器完成排序和分组,未指定具体的排序和分组比较器。
MapTask&ReduceTask使用用户自定义的排序比较器完成排序和分组，未指定分组比较器，但指定了排序比较器
MapTask&ReduceTask使用用户自定义的排序比较器进行排序，reduce使用自定义的分组比较器进行分组。既指定的排序比较器，有指定了分组比较。
MapTask使用key自带的比较器，ReduceTask使用用户自定义的分组比较。

如何设置分组比较器？

job.setGroupingComparatorClass(WCGroupingComparator.class);

设置分组比较器底层代码：

public void setGroupingComparatorClass(Class<? extends RawComparator> cls
                                       ) throws IllegalStateException {
  ensureState(JobState.DEFINE);
  conf.setOutputValueGroupingComparator(cls);
}

setOutputValueGroupingComparator():

public void setOutputValueGroupingComparator(
      Class<? extends RawComparator> theClass) {
    setClass(JobContext.GROUP_COMPARATOR_CLASS,
             theClass, RawComparator.class);
  }

JobContext.GROUP_COMPARATOR_CLASS就和获取使用使用的同一个常量。

如何自定义分组比较器？

package com.itbaizhan;

import org.apache.hadoop.io.RawComparator;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;

public class WCGroupingComparator extends WritableComparator {
    public WCGroupingComparator(){
        super(KeyClass.class);
    }
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        //编写比较器的比较逻辑
        return super.compare(a, b);
    }
}

1.5.4 Reduce类

在ReduceTask的run方法中：

if (useNewApi) {//hadoop2.x+ 
    runNewReducer(job, umbilical, reporter, rIter, comparator, 
                  keyClass, valueClass);
} else {//hadoop1.x
    runOldReducer(job, umbilical, reporter, rIter, comparator, 
                  keyClass, valueClass);
}

runNewReducer():

 private <INKEY,INVALUE,OUTKEY,OUTVALUE>
  void runNewReducer(JobConf job,
                     final TaskUmbilicalProtocol umbilical,
                     final TaskReporter reporter,
                     RawKeyValueIterator rIter,
                     RawComparator<INKEY> comparator,
                     Class<INKEY> keyClass,
                     Class<INVALUE> valueClass
                     ) throws IOException,InterruptedException, 
                              ClassNotFoundException {
    // wrap value iterator to report progress.
    final RawKeyValueIterator rawIter = rIter;
    rIter = new RawKeyValueIterator() {
      public void close() throws IOException {
        rawIter.close();
      }
      public DataInputBuffer getKey() throws IOException {
        return rawIter.getKey();
      }
      public Progress getProgress() {
        return rawIter.getProgress();
      }
      public DataInputBuffer getValue() throws IOException {
        return rawIter.getValue();
      }
      public boolean next() throws IOException {
        boolean ret = rawIter.next();
        reporter.setProgress(rawIter.getProgress().getProgress());
        return ret;
      }
    };
    // make a task context so we can get the classes
    org.apache.hadoop.mapreduce.TaskAttemptContext taskContext =
      new org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl(job,
          getTaskID(), reporter);
    // make a reducer  创建Reduce实例
    org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE> reducer =
      (org.apache.hadoop.mapreduce.Reducer<INKEY,INVALUE,OUTKEY,OUTVALUE>)
        ReflectionUtils.newInstance(taskContext.getReducerClass(), job);
    org.apache.hadoop.mapreduce.RecordWriter<OUTKEY,OUTVALUE> trackedRW = 
      new NewTrackingRecordWriter<OUTKEY, OUTVALUE>(this, taskContext);
    job.setBoolean("mapred.skip.on", isSkipping());
    job.setBoolean(JobContext.SKIP_RECORDS, isSkipping());
    org.apache.hadoop.mapreduce.Reducer.Context 
         reducerContext = createReduceContext(reducer, job, getTaskID(),
                                               rIter, reduceInputKeyCounter, 
                                               reduceInputValueCounter, 
                                               trackedRW,
                                               committer,
                                               reporter, comparator, keyClass,
                                               valueClass);
    try {
      reducer.run(reducerContext);
    } finally {
      trackedRW.close(reducerContext);
    }
  }

通过taskContext.getReducerClass()获取自定义Reducer类的Class对象，然后在通过反射ReflectionUtils.newInstance()创建对应的实例。

getReducerClass():

public Class<? extends Reducer<?,?,?,?>> getReducerClass() 
    throws ClassNotFoundException;

Ctrl+Alt+B->JobContextImpl类的getReducerClass()方法：

public Class<? extends Reducer<?,?,?,?>> getReducerClass() 
    throws ClassNotFoundException {
    return (Class<? extends Reducer<?,?,?,?>>) 
        conf.getClass(REDUCE_CLASS_ATTR, Reducer.class);
}

先根据常量REDUCE_CLASS_ATTR对应的字符串找配置文件中的value对应的Class，如果没有找到自定义的Reducer类，则使用默认的Reducer类。

如何设置自定义的Reducer类呢？

job.setReducerClass(WCReducer.class);

底层是如何对应的？

public void setReducerClass(Class<? extends Reducer> cls
                           ) throws IllegalStateException {
    ensureState(JobState.DEFINE);
    conf.setClass(REDUCE_CLASS_ATTR, cls, Reducer.class);
}

设置时也是通过常量REDUCE_CLASS_ATTR进行设置的，所以读取的就是我们设置的自定义Reducer类。

1.5.5 Reducer运行时run方法

ReducerTask类的runNewReducer()方法中：

private <INKEY,INVALUE,OUTKEY,OUTVALUE> void runNewReducer(...){
    ......
    try {
      reducer.run(reducerContext);
    } finally {
      trackedRW.close(reducerContext);
    }
    ......
}

reducer.run(reducerContext)：

public void run(Context context) throws IOException, InterruptedException {
    //同一个ReduceTask会执行一次setup方法
    setup(context);
    try {
        while (context.nextKey()) {
            //同一组数据调用一次reduce方法，如果指定了自定义的Reducer类，就调用自定义类中的reduce方法
            reduce(context.getCurrentKey(), context.getValues(), context);
            // If a back up store is used, reset it
            Iterator<VALUEIN> iter = context.getValues().iterator();
            if(iter instanceof ReduceContext.ValueIterator) {
                ((ReduceContext.ValueIterator<VALUEIN>)iter).resetBackupStore();        
            }
        }
    } finally {
        //同一个ReduceTask会执行一次cleanup方法
        cleanup(context);
    }
}

context.nextKey()

public boolean nextKey() throws IOException,InterruptedException;

Ctrl+Alt+B->ReduceContextImpl:

public boolean nextKey() throws IOException,InterruptedException {
    while (hasMore && nextKeyIsSame) {
      nextKeyValue();
    }
    if (hasMore) {
      if (inputKeyCounter != null) {
        inputKeyCounter.increment(1);
      }
      //还有下一组数据
      return nextKeyValue();
    } else {
      //后续没有下一组数据则返回false
      return false;
    }
  }

nextKeyValue():

//进入reduce方法时，“相同的”key为一组，调用一次reduce方法
public boolean nextKeyValue() throws IOException, InterruptedException {
    //没有下一组数据，直接将key和value置空，并返回false
    if (!hasMore) {
      key = null;
      value = null;
      return false;
    }
    //还有下一组数据则继续
    firstValue = !nextKeyIsSame;
    //获取到key
    DataInputBuffer nextKey = input.getKey();
    currentRawKey.set(nextKey.getData(), nextKey.getPosition(), 
                      nextKey.getLength() - nextKey.getPosition());
    buffer.reset(currentRawKey.getBytes(), 0, currentRawKey.getLength());
    key = keyDeserializer.deserialize(key);
    //获取value
    DataInputBuffer nextVal = input.getValue();
    buffer.reset(nextVal.getData(), nextVal.getPosition(), nextVal.getLength()
        - nextVal.getPosition());
    value = valueDeserializer.deserialize(value);

    currentKeyLength = nextKey.getLength() - nextKey.getPosition();
    currentValueLength = nextVal.getLength() - nextVal.getPosition();

    if (isMarked) {
      backupStore.write(nextKey, nextVal);
    }

    hasMore = input.next();
    if (hasMore) {
      nextKey = input.getKey();
      nextKeyIsSame = comparator.compare(currentRawKey.getBytes(), 0, 
                                     currentRawKey.getLength(),
                                     nextKey.getData(),
                                     nextKey.getPosition(),
                                     nextKey.getLength() - nextKey.getPosition()
                                         ) == 0;
    } else {
      nextKeyIsSame = false;
    }
    inputValueCounter.increment(1);
    return true;
  }

context.getCurrentKey()

public KEYIN getCurrentKey() throws IOException, InterruptedException;

Ctrl+Alt+B->ReduceContextImpl:

public KEYIN getCurrentKey() {
    return key;
}

获取是在nextKeyValue方法中赋的值。

context.getValues()

public Iterable<VALUEIN> getValues() throws IOException, InterruptedException;

Ctrl+Alt+B->ReduceContextImpl:

public Iterable<VALUEIN> getValues() throws IOException, InterruptedException {
    return iterable;
}

context.write()

public void write(KEYOUT key, VALUEOUT value) 
      throws IOException, InterruptedException;

Ctrl+Alt+B->ChainReduceContextImpl:

  public void write(KEYOUT key, VALUEOUT value) throws IOException,
      InterruptedException {
    rw.write(key, value);
  }

点击write方法：

  /**Writes a key/value pair.
   * @param key the key to write.
   * @param value the value to write.
   * @throws IOException
   */      
  public abstract void write(K key, V value
                             ) throws IOException, InterruptedException;

Ctrl+Alt+B->LineRecordWriter:

public synchronized void write(K key, V value)
      throws IOException {
    boolean nullKey = key == null || key instanceof NullWritable;
    boolean nullValue = value == null || value instanceof NullWritable;
    if (nullKey && nullValue) {
        //如果key和value均为null或NullWritable,直接退出
        return;
    }
    if (!nullKey) {//key不为空
        writeObject(key);
    }
    if (!(nullKey || nullValue)) {//key和value都不为空，输出分隔符 :\t
        out.write(keyValueSeparator);
    }
    if (!nullValue) {//value不为空
        writeObject(value);
    }
    out.write(NEWLINE);//换行
}

key和value之间的分隔符默认为\t

public LineRecordWriter(DataOutputStream out) {
    this(out, "\t");
}

如何设置自定义的OutputFormat类：

job.setOutputFormatClass(Xxx.class);

留不住的人

关注

1
点赞
踩
1

收藏

觉得还不错? 一键收藏
打赏
0
评论
Hadoop-MapReduce源码分析

#一、MapReduce源码分析在WCMapper类中的map方法的首行添加如下代码：重新打jar包上传到hadoop集群中，重新运行在hadoop集群中的任何节点上执行如下命令：下载文件夹/tmp/hadoop-yarn/staging/root/.staging/job_1635443832663_0002从该节点的/root目录下下载到windows系统的桌面上，内容列表如下将job.xml拷贝到wordcount项目的根目录，然后进行格式化（目的：方便查看参数），使用C
复制链接

扫一扫