mapred.map.tasks 如何影响map的个数

且具体到底产生多少个分片(split) 因为多少个map 是有关系。(此处是根据新的API来分析，因为新的API 终究要调用到就得API来做具体的动作)

可能会说这个值 是系统根据文件大小 和根据文件分片大小 算出来的，那具体是如何算出来的呢，我们根据源码 一步一步来分析

首先Job.submit()

public void submit() throws IOException, InterruptedException,
ClassNotFoundException {
ensureState(JobState.DEFINE);
setUseNewAPI();

// Connect to the JobTracker and submit the job
connect();
info = jobClient.submitJobInternal(conf); //此处用到JobClient的submitJobInternal 方法 看下面源码2

super.setJobID(info.getID());
state = JobState.RUNNING;
}

源码2 JobClient.submitJobInternal()

我们看 此方法中的 下面一段

// Create the splits for the job
FileSystem fs = submitJobDir.getFileSystem(jobCopy);
LOG.debug("Creating splits at " + fs.makeQualified(submitJobDir));
int maps = writeSplits(context, submitJobDir);//在此处计算 具体有多少个map紧接着看下面源码3
jobCopy.setNumMapTasks(maps);

源码3 writeSplits()

private int writeSplits(org.apache.hadoop.mapreduce.JobContext job,
Path jobSubmitDir) throws IOException,
InterruptedException, ClassNotFoundException {
JobConf jConf = (JobConf)job.getConfiguration();
int maps;
if (jConf.getUseNewMapper()) {//新API
maps = writeNewSplits(job, jobSubmitDir); //见下面源码5
} else {//旧API
maps = writeOldSplits(jConf, jobSubmitDir);
}
return maps;
}

源码5 writeNewSplits()

int writeNewSplits(JobContext job, Path jobSubmitDir) throws IOException,
InterruptedException, ClassNotFoundException {
Configuration conf = job.getConfiguration();
InputFormat<?, ?> input =
ReflectionUtils.newInstance(job.getInputFormatClass(), conf);

List<InputSplit> splits = input.getSplits(job); //我们需要关注的是这一行 调用input 中的getSplits 方法 我们会用FileInputFormat的getSplits方法来做实例 看源码6
T[] array = (T[]) splits.toArray(new InputSplit[splits.size()]);

// sort the splits into order based on size, so that the biggest
// go first
Arrays.sort(array, new SplitComparator());
JobSplitWriter.createSplitFiles(jobSubmitDir, conf,
jobSubmitDir.getFileSystem(conf), array);
return array.length;
}

源码6 FileInputFormat.getSplits(JobConf job, int numSplits)

FileStatus[] files = listStatus(job);

// Save the number of input files in the job-conf
job.setLong(NUM_INPUT_FILES, files.length);
long totalSize = 0; // compute total size
for (FileStatus file: files) { // check we have valid files
if (file.isDir()) {
throw new IOException("Not a file: "+ file.getPath());
}
totalSize += file.getLen();
}

long goalSize = totalSize / (numSplits == 0 ? 1 : numSplits);
long minSize = Math.max(job.getLong("mapred.min.split.size", 1),
minSplitSize);

// generate splits
ArrayList<FileSplit> splits = new ArrayList<FileSplit>(numSplits);
NetworkTopology clusterMap = new NetworkTopology();
for (FileStatus file: files) {
Path path = file.getPath();
FileSystem fs = path.getFileSystem(job);
long length = file.getLen();
BlockLocation[] blkLocations = fs.getFileBlockLocations(file, 0, length);
if ((length != 0) && isSplitable(fs, path)) {
long blockSize = file.getBlockSize();
long splitSize = computeSplitSize(goalSize, minSize, blockSize);//此处计算 split size 从这可以看出来 是根据goal,minSize,blockSize 三个参数来计算的，那需要仔细看看上面三个参数的由来，再继续看源码 7

long bytesRemaining = length;
while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
String[] splitHosts = getSplitHosts(blkLocations,
length-bytesRemaining, splitSize, clusterMap);
splits.add(new FileSplit(path, length-bytesRemaining, splitSize,
splitHosts));
bytesRemaining -= splitSize;
}

if (bytesRemaining != 0) {
splits.add(new FileSplit(path, length-bytesRemaining, bytesRemaining,
blkLocations[blkLocations.length-1].getHosts()));
}
} else if (length != 0) {
String[] splitHosts = getSplitHosts(blkLocations,0,length,clusterMap);
splits.add(new FileSplit(path, 0, length, splitHosts));
} else {
//Create empty hosts array for zero length files
splits.add(new FileSplit(path, 0, length, new String[0]));
}
}
LOG.debug("Total # of splits: " + splits.size());
return splits.toArray(new FileSplit[splits.size()]);

源码7 computeSplitSize

protected long computeSplitSize(long goalSize, long minSize,
long blockSize) {
return Math.max(minSize, Math.min(goalSize, blockSize)); 具体计算公式如下，可以看出
}

所以从上面源码的解读，我们可以看出来，这个参数 mapred.map.tasks 的设置对具体的mapreduce 对输入进行分片产生一定的作用，因为具体产生多少分片，多少个map

是根据三个参数来决定的 一个是dfs.block.size 另外一个是mapred.map.tasks 还有一个 就是 mapred.min.split.size 但一般情况下如果不设置 mapred.map.tasks 的情况下 则会根据其它两个参数来决定，但一般情况下 mapred.min.split.size 参数我们也不设置，所以 dfs.block.size 自然就是我们默认的分片大小，如果mapred.min.split.size 大于dfs.block.size 则系统分片就会大于文件系统 块的大小，从而map的个数也会相应的减少。