mapred.map.tasks 如何影响map的个数_mapred.map.tasks参数值-CSDN博客

且具体到底产生多少个分片(split) 因为多少个map 是有关系。(此处是根据新的API来分析，因为新的API 终究要调用到就得API来做具体的动作)

可能会说这个值是系统根据文件大小和根据文件分片大小算出来的，那具体是如何算出来的呢，我们根据源码一步一步来分析

首先Job.submit()

public void submit() throws IOException, InterruptedException,
                              ClassNotFoundException {
    ensureState(JobState.DEFINE);
    setUseNewAPI();

    // Connect to the JobTracker and submit the job
    connect();
    info = jobClient.submitJobInternal(conf); //此处用到JobClient的submitJobInternal 方法看下面源码2

    super.setJobID(info.getID());
    state = JobState.RUNNING;
   }

源码2 JobClient.submitJobInternal()

我们看此方法中的下面一段

// Create the splits for the job
          FileSystem fs = submitJobDir.getFileSystem(jobCopy);
          LOG.debug("Creating splits at " + fs.makeQualified(submitJobDir));
          int maps = writeSplits(context, submitJobDir);  //在此处计算具体有多少个map 紧接着看下面源码3
          jobCopy.setNumMapTasks(maps);

源码3 writeSplits()

private int writeSplits(org.apache.hadoop.mapreduce.JobContext job,
      Path jobSubmitDir) throws IOException,
      InterruptedException, ClassNotFoundException {
    JobConf jConf = (JobConf)job.getConfiguration();
    int maps;
    if (jConf.getUseNewMapper()) {//新API
      maps = writeNewSplits(job, jobSubmitDir); //见下面源码5
    } else {//旧API
      maps = writeOldSplits(jConf, jobSubmitDir);
    }
    return maps;
}

源码5 writeNewSplits()

int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
      InterruptedException, ClassNotFoundException {
    Configuration conf = job.getConfiguration();
    InputFormat<?, ?> input =
      ReflectionUtils.newInstance(job.getInputFormatClass(), conf);

List<InputSplit> splits = input.getSplits(job); //我们需要关注的是这一行调用input 中的getSplits 方法我们会用FileInputFormat的getSplits方法来做实例看源码6
T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);

    // sort the splits into order based on size, so that the biggest
    // go first
    Arrays.sort(array, new SplitComparator());
    JobSplitWriter.createSplitFiles(jobSubmitDir, conf,
        jobSubmitDir.getFileSystem(conf), array);
    return array.length;
}

源码6 FileInputFormat.getSplits(JobConf job, int numSplits)

FileStatus[] files = listStatus(job);

    // Save the number of input files in the job-conf
    job.setLong(NUM_INPUT_FILES, files.length);
    long totalSize = 0;                           // compute total size
    for (FileStatus file: files) {                // check we have valid files
      if (file.isDir()) {
        throw new IOException("Not a file: "+ file.getPath());
      }
      totalSize += file.getLen();
    }

    long goalSize = totalSize / (numSplits == 0 ? 1 : numSplits);
    long minSize = Math.max(job.getLong("mapred.min.split.size", 1),
                            minSplitSize);

    // generate splits
    ArrayList<FileSplit> splits = new ArrayList<FileSplit>(numSplits);
    NetworkTopology clusterMap = new NetworkTopology();
    for (FileStatus file: files) {
      Path path = file.getPath();
      FileSystem fs = path.getFileSystem(job);
      long length = file.getLen();
      BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0, length);
      if ((length != 0) && isSplitable(fs, path)) {
        long blockSize = file.getBlockSize();
        long splitSize = computeSplitSize(goalSize, minSize, blockSize);//此处计算 split size 从这可以看出来是根据goal,minSize,blockSize 三个参数来计算的，那需要仔细看看上面三个参数的由来，再继续看源码 7

        long bytesRemaining = length;
        while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
          String[] splitHosts = getSplitHosts(blkLocations,
              length-bytesRemaining, splitSize, clusterMap);
          splits.add(new FileSplit(path, length-bytesRemaining, splitSize,
              splitHosts));
          bytesRemaining -= splitSize;
        }

        if (bytesRemaining != 0) {
          splits.add(new FileSplit(path, length-bytesRemaining, bytesRemaining,
                     blkLocations[blkLocations.length-1].getHosts()));
        }
      } else if (length != 0) {
        String[] splitHosts = getSplitHosts(blkLocations,0,length,clusterMap);
        splits.add(new FileSplit(path, 0, length, splitHosts));
      } else {
        //Create empty hosts array for zero length files
        splits.add(new FileSplit(path, 0, length, new String[0]));
      }
    }
    LOG.debug("Total # of splits: " + splits.size());
    return splits.toArray(new FileSplit[splits.size()]);

源码7 computeSplitSize

protected long computeSplitSize(long goalSize, long minSize,
long blockSize) {
return Math.max(minSize, Math.min(goalSize, blockSize)); 具体计算公式如下，可以看出
}

所以从上面源码的解读，我们可以看出来，这个参数 mapred.map.tasks 的设置对具体的mapreduce 对输入进行分片产生一定的作用，因为具体产生多少分片，多少个map

是根据三个参数来决定的一个是dfs.block.size 另外一个是mapred.map.tasks 还有一个就是 mapred.min.split.size 但一般情况下如果不设置 mapred.map.tasks 的情况下则会根据其它两个参数来决定，但一般情况下 mapred.min.split.size 参数我们也不设置，所以 dfs.block.size 自然就是我们默认的分片大小，如果mapred.min.split.size 大于dfs.block.size 则系统分片就会大于文件系统块的大小，从而map的个数也会相应的减少。

文章来自：http://blog.csdn.net/zhouleilei/article/details/7836529